Deceptive Alignment Monitoring

The use of foundation models to generate and/or curate their own training data has recently emerged in both language and vision. In language generation, Self-Instruct used one language model to bootstrap training data for fine-tuning a different language model (Wang et al., 2022), and similar methods have quickly become commonplace, e.g., (Peng et al., 2023; Sun et al., 2023; Xu et al., 2023). In curation, SemDeDup uses an high-performing language model to prune a dataset for training future language models (Abbas et al., 2023), and DoReMi learns to weight data differently to improve performance when learning (Xie et al., 2023). In vision, (Li et al., 2023) propose endowing a model with the ability to learn to construct its own training dataset in a self-supervised manner by trawling through the internet. Such methods will continue to improve and proliferate.

These methods are relevant to deceptive alignment because if a highly capable, autonomous model does wish to change its behavior, creating or curating training data are straightforward mechanisms by which it can achieve such a goal. Currently, such methods have a high degree of human-in-the-loop, but as model scale and complexity increase, humans will play a vanishing role. To the best of our knowledge, we lack methods for monitoring the training data that our models generate or curate to check that the training data has not been selected in pursuit of some ulterior goal. This threat is spiritually similar to recent work demonstrating that web-scale datasets can be poisoned (Carlini et al., 2023); however, in the setting studied by Carlini et al. (2023), the dataset is known, fixed and constructed by humans, whereas in this setting, the dataset is unknown, dynamic and constructed by models in a largely unsupervised manner.

Turning now from data to learning, there are two emerging trends that pose new challenges: using one model to train another model, and directly editing the parameters of a model to change its behavior. The first trend appears commonly when using reward or preference models to fine-tune a language model, such as in Reinforcement Learning from Human Feedback (Ouyang et al., 2022; Bai et al., 2022a), Reinforcement Learning from AI Feedback (Bai et al., 2022b), Reinforcement Learning for Feedback (Akyürek et al., 2023) and many more, e.g., image synthesis (Lee et al., 2023). The second trend includes several methods for model editing, i.e., directly editing the model’s parameters to change its behavior on specific tasks or inputs, e.g., Meng et al. (2022a); Mitchell et al. (2022); Meng et al. (2022b); Hartvigsen et al. (2022); Ravfogel et al. (2022); Hoelscher-Obermaier et al. (2023). In some methods, models are directly trained to learn to modify their own parameters, e.g., Irie et al. (2022); Kirsch & Schmidhuber (2022b, a).

Why does this matter for deceptive alignment? If models can have ulterior motives and act on said motives, then enabling models to train each other or to edit themselves directly posses a significant threat, since a model can give itself or give another model capabilities that humans would rather the model(s) not possess. To the best of our knowledge, we lack techniques to audit what learning signals one model transmits to another model, and we also lack techniques to understand whether model editing methods can themselves be poisoned; for a concrete example, it might be possible that by selective choosing a small number of seemingly innocent edits (e.g., “US President: Donald Trump $\rightarrow$ Joe Biden” and “World Chess Champion: Magnus Carlsen $\rightarrow$ Ding Liren”), the combination of edits induces an unpredictable and undesirable change in model behavior¹¹1However, we note that sometimes seemingly unpredictable changes are indeed predictable, e.g., Schaeffer et al. (2023)..

A key step towards addressing the identified challenges will be monitoring the representations and mechanisms of large-scale models for “permissible” behavior (Christiano, 2022a, b); here, “permissible” behavior is loosely understood since the concept itself is in desperate need of sharper understanding. Loosely, if we think of a network as some map $f:\mathcal{X}\rightarrow\mathcal{Y}$, previous work has studied adversarial inputs coming from an unusual input distribution $\tilde{p}(x)$ or backdoored models with unusual conditional distributions $p(y|x)$. We are therefore concerned not with unusual input data, nor unusual conditional distributions, but whether the network’s outputs are produced for the “normal reasons”. As Christiano (2022a) explains, we are looking for outliers in terms of how $f$ handles perfectly normal data.

The reason why this matters is that while inputs, outputs, and inputs-to-output maps may all appear perfectly ordinary, highly capable models may conceal some ulterior motive or goal. In order to detect and counter this threat, it is imperative to develop interpretability methods capable of understanding both activations and circuit mechanisms of models and detecting abnormal variations in models’ processing. This is important because two models generating identical outputs when provided with the same inputs may compute their outputs for different “reasons”.

Because the exact threat is unknown and likely dynamic, there is an urgent need to develop unsupervised methods for mechanistic anomaly detection that scale well. The next step is to develop an unsupervised methodology that can differentiate between normal model behaviors—where conclusions are reached for “appropriate reasons”—and anomalous model behaviors—where conclusions are derived for erroneous, potentially harmful “reasons”. This task is particularly challenging since it requires the identification of patterns in the propagation of information through a model that is indicative of correct reasoning without relying on explicitly supervisory signals.

To achieve this, we propose leveraging techniques related to unsupervised anomaly detection to capture deviations from typical model behaviors. By comparing a model’s processing across various inputs and outputs, it may be possible to identify patterns that consistently align with desired and appropriate behavior. We hypothesize that these patterns could manifest at three different levels of analysis within a model. Firstly, at the individual layer, a comprehensive analysis of activation distributions in the high-dimensional activation space could provide valuable insights into the model’s processing. Secondly, at the layer-to-layer activation level, investigating how high-dimensional modes propagate, transform and evolve through the layers of a model can also offer an understanding of normal and abnormal processing. Thirdly, at the circuit level, identifying subgraphs within the network that correspond to specific transformations on features relevant to out-of-domain generalization might also prove powerful; however, knowing how to usefully define probabilistic distribution over activations, activations’ propagations and circuit mechanisms for anomaly detection are, to the best of our knowledge, open questions. For possible approaches, see Carranza et al. (2023).