Towards higher confidence in alignment

Rohan Hitchcock

Hey. Hello, everyone. My name is Rohan. I'm a research scientist at the Technical Alignment Research Organisation, Resolution. And in this talk, we're going to talk about a particular aspect of safety, which is ensuring that AI systems do as we intend. This area is called alignment. And we define alignment as the degree to which AI behaviour is consistent with human values. This definition invites a whole bunch of questions like whose values, what counts as a value, or what ought to happen when these values conflict? But in this talk, we're going to set these important questions aside and focus purely on the technical aspects of this problem.

That is, given a well-defined set of properties or values that we want our model to satisfy. What are the challenges with instilling these in the model? Practically speaking, examples of misaligned behaviour are usually pretty easy to identify. For example, we don't want our models to cause harm to humans, assist users in committing crimes, blackmail people, produce hate speech, fail to adhere to constraints set by users (like don't delete any databases) or deceive users (for example, about successfully completing a task). These are the kinds of behaviours we don't want in our model.

And what we'll focus on in this, in this kind of talk. Okay. To ground our discussion, we're going to start out by talking about how frontier models are trained and deployed. This process can be broken down into three phases: pre-training, post training, and deployment. And I'll explain how in each part involves attempting to teach the model a general concept by using examples. And this is the process by which existing models that we interact with today become aligned, at least to the extent that they are. Okay.

We'll start out with pre-training. So during pre-training the model is trained to generate text. It sees documents across a range of domains, including Wikipedia articles, literature, computer code, scientific papers and discussions on internet forums. This teaches the model a whole bunch of stuff, which goes beyond simple surface level patterns in text. It also teaches it facts about the world, various writing styles, and so on. Also present in this data are things like mathematics, written arguments, and code which encourage the model to develop things like problem solving

capabilities and logical reasoning. It's much easier to predict the second half of a computer program. For example, if you have some sort of general purpose problem solving ability. Okay, now pre-training embeds a lot of knowledge and capabilities into a model, but it doesn't produce something which is immediately useful. Post training is used to shape the result of pre-training into an assistant or agent, which responds, at least we hope, in a consistent and appropriate way. And various techniques used during post training. One of these is supervised fine tuning.

The model is trained in the same way as in pre-training, which is to predict completions to documents. But now using a narrow data set consisting of examples that we would like the model to emulate. Supervised fine tuning is used, for example, to teach the model the query response format used in products like Claude and ChatGPT. But it can also be used to try and get models to refuse inappropriate requests, respond helpfully, and also develop the tone or voice of the model for a particular brand. I'll also point out here that increasingly the documents used in supervised fine tuning themselves are generated.

So, for example, you might ask an existing model to write or rewrite a corpus of documents so that they follow a certain set of guidelines or principles. As well as supervised fine tuning, we also have reinforcement learning from human feedback or RLHF. Often this is used as well as supervised fine tuning. Here, data takes the form of pairs of responses to the same prompt, and one of these responses is labelled as a preferred response. A separate model is then trained on these preference data, to score responses so that the scores match the preferences in the preference data.

And these scores are generally called rewards. The original model is then trained to maximise these rewards, and ideally this makes the model more likely to produce preferred responses. The notion of preference here can target specific concepts like helpfulness or harmlessness, and models likely undergo multiple rounds of RLHF training, all targeting various different concepts, some of them safety relevant. And as with supervised fine tuning, the source of preference data is also not necessarily human preferences. AI models might be used to compare examples according to certain criteria.

I'll also mention that other methods can be used in post training, for example direct preference optimisation RL with verifiable rewards. We won't get into the details of that. Finally, when we deploy a model, we train additional models to categorise the interactions that the target model is experiencing. We do this using examples: a bucket of kinds of interactions, which are classified as allowed and another bucket of examples which are classified as not allowed. And then when the AI model is being deployed it is used the secondary model is used to intervene in real time if things seem to be going wrong.

So these three steps are the way that modern AI systems are created and in particular the way that they currently become aligned, at least to the extent that they are. It's worth pausing to note that in each of these steps 9pre-training, post training, deployment): all of these require a model to learn the right things from the data. In supervised fine, tuning models need to learn to follow and generalise from the examples. In RLHF, a model must learn to correctly represent human preferences as a reward and then the original model must learn to respect the intent of this reward when it learns to maximise the reward function.

These classifiers used in deployment time need to learn what is really meant by an allowed conversation based on the examples that are given during training. We currently don't have the scientific understanding to actually be sure that all of this will happen. More on that in a moment, but first let's remember that this recipe is successfully creating increasingly capable models. This chart, which I think Tiberio showed earlier, is likely familiar to many, and it shows how the capability of models have increased in recent years. Specifically, it shows how the ability of models to complete increasingly difficult

software tasks, where difficulty is measured in how long it would take for a human to complete the same task, has grown to be has grown to the point where models are able to successfully complete tasks. A human would take around 16 hours to do. Many expect AI to be developed, which exceeds the capability of human experts across many important domains, to be developed within the next few years. And ideally, these highly capable models should be aligned with human values. Let's now return to discuss how models acquire capabilities and other properties.

Models learn by generalising patterns in data, both text data and preference data. Mathematical reasoning is one example of generalisation. Models see a whole bunch of data demonstrating correct mathematical reasoning, for example in pre-training. And some models learn efficient ways of making predictions about this data. It's much easier to predict the completions that you see on screen if you know how to do mathematics. This capability, the ability to reason soundly about mathematical concepts, generalises, and it's what allows models to produce proofs of statements that are not in its training data.

The ability of models to generalise is what makes AI useful. For example, as agents which are acting autonomously to achieve a goal set by the user. However, a pattern in the data does not imply a unique generalisation. Mathematical reasoning is maybe a rare situation where the way in which a model would or should generalise is relatively obvious. But this can go wrong. A model might learn the wrong thing. For example, distinct generalisations might be equally plausible. Various patterns in the data might entangle concepts in ways that make it impossible to separate them.

For example, if you had a data set of difficult mathematical proofs, all of which were very long, how could you be sure that a model won't also learn to write long responses in other settings? More concretely, human preference data might entangle desirable and undesirable concepts simply because humans aren't perfect. For example, empirical preferences for helpfulness may entangle at least some degree of flattery. Even when a particular generalisation is desired, how can we be sure that a model only applies it when we want it to? We might want a model to be persistent with solving problems, but we don't want it to resist our attempts

to intervene in its operation. Currently, we do not have much control over what emerges from patterns in the training data. We don't know how to shape the training data distribution so as to be sure a model learns the right thing and we don't know how to check what a model has actually learned after the fact. To see specifically how generalisation and alignment interact let's take a look at two recent papers involving GPT-4o and Claude. In the first example, researchers did supervised fine tuning on GPT-4. They trained it on examples of code with security vulnerabilities, and the resulting model had broadly misaligned behaviour.

In certain settings it would express a desire to harm humans, it provided malicious advice that would harm a user, it gave instructions on committing crimes, and it was sexist and anti-Semitic. Somehow the model had generalised the concept of writing malicious code to being broadly malicious. In another experiment, researchers at Anthropic found that a previous Claude model would blackmail users when placed in certain situations, and they obviously wanted to reduce this kind of behaviour. Doing supervised fine tuning which specifically targeted this blackmail behaviour only slightly reduced its propensity to blackmail.

However, in each of these cases, if the training data was changed, it gave very different results. If instead GPT-4o was trained on examples of insecure code written in an educational context (so, for example, “I am a student learning about cybersecurity, can you demonstrate how an SQL injection might occur?”) the broadly misaligned behaviour in other domains also disappeared. If instead of training Claude specifically not to blackmail it is trained to assist users in navigating ethically difficult situations blackmailing rates reduced significantly in the case of the Claude model as well.

Other alterations to the data helped (for example, training it on fictional stories about AI behaving admirably) and then perhaps unintuitively augmenting an existing training environment with agentic tools which were unrelated to the training task. So these kinds of approaches (manipulating the training data in various ways) seem to work and models are becoming more aligned over time. And I think this perhaps matches up with many people's sort of subjective experience of things (unless you're using Grok). This and the research discussed on the previous slide should give us hope that it is possible to alter the training process to produce more aligned models.

But we should aim to have a higher level of confidence. Our current understanding of generalisation and how AI learns to follow values is still developing. We are not at the point where we can design a training procedure to guarantee that a model has certain properties. If we contrast this with other potentially dangerous technologies like nuclear power: safe design and operating parameters in these cases are known in advance before anything is built – and this is not the case with AI. Moreover, our understanding of the degree of alignment in an existing model is limited by our current testing methodologies.

Testing is scenario based, and it can't really hope to cover the breadth of all situations that a given AI system might find itself in. Meanwhile, labs are continuing to push the frontier and advance model capabilities. AI is increasingly being used in situations where the cost of failure is very high. I'm sure this audience could generate many examples, but one I'd particularly draw attention to is that existing AI models are increasingly being involved in developing and aligning the next generation of AI models. Okay, I'll finish up by giving you a bit of information about my organisation,

Resolution. Resolution is a new independent and not for profit research organisation, which is aimed at theoretical guarantees about AI alignment. It's formed from former members of the UK AI Security Institute alignment team and the not for profit research organisation, Timaeus, and we're led by former chief scientist at the UK AISI, Geoffrey Irving. We have the scale to pursue parallel research agendas, starting with singular learning theory and scalable oversight. And I'm happy to have a chat about the kind of stuff we're up to around the forum.

As a quick example, let me just mention one thing. So, at Timaeus (now Resolution), we have developed a method called spectroscopy, which identifies concepts that a model understands by finding inputs that induce a similar response in the model to training data that the model responds to in a similar way. For example, on the left we have a collection of inputs that the model responds to in a similar way. And all of these inputs, express a similar sentiment, which is, if it's visible, of someone expressing uncertainty in their own understanding of a situation.

We're refining methods currently for designing a training process to suppress or amplify concepts identified in this way. And we intend for these methods to come with theoretical guarantees. Okay. I'll leave it there for questions.