AI Character Drift Under Automated R&D

Thank you. Talking about character drift in automated AI R&D. I'm Slava. I've spent the last couple of years mostly working on interpretability. This is not an interpretability talk. And recently I've joined Lyptus Research, which is a new organisation here in Australia doing technical AI research. Also, we have Jack here in the audience who has done most of the work for this research. Got to have this slide. Here is an outline. I'm going to present not one but two methodology sections, with two sets of results. But first some background.

Today already models are helping in the development of the next generation of AI models. They help in a bunch of ways. I've listed some here. So for example, they help with data curation, RL from AI feedback. Instead of the human we have an AI grading AI responses, grading evals. Obviously these days the AI agents that write the code and as AI capabilities improve, we have increasingly agentic systems [inaudible] going off and running their own experiments and coming back with some sort of results. So far, what we have is the current generation of models is helping train the next generation.

So model N helps train model N plus one, and as [inaudible] as these models get more capable, they play a larger and larger part in the training of the next generation. And it's a good question as to where this goes [inaudible] when the next model is used to train the model after that, and the one after that, and so on. So what happens to values? What happens to AI character? I think there's a pretty interesting question. So, model character. Character is how the model tends to act in various situations and what [inaudible] what values it has.

So I have a pretty broad view of what character is — just how it responds to users. And character is very important. And for example, a lot of work goes into Claude's character, making Claude Claude-y. And I think that's part of why I enjoy interacting with Claude. And I trust it. So its character is probably a large part of why I trust it to go out as an agent, doing my bidding for me and making reasonable moves without being supervised. As we know, this also can go pretty wrong.

So most of you will be familiar with the GPT-4o sycophancy incident where it became really sycophantic. It would reinforce the various beliefs and biases of its users. Potentially some people got AI psychosis, and I would argue that character gets more important as time goes on and as capabilities improve, because the models tend to act more agentically under less supervision. So it really matters what fundamental character they have when you're not observing their actions. Now, character isn't just a spec written into the system prompt of the model you're interacting with.

It's actually trained. It's part of the weights of the model, and it's trained during the training phase. And as we saw, training is increasingly happening with the help of AI models. So it's a very obvious question to ask, how do these two things interact? How will character evolve under automated AI R&D? Will character traits and values be preserved, passed down through the generations of models, or will they drift? Can we study this somehow? It seems pretty important. But it's also difficult to get traction on as a research problem.

So this presentation is about our first attempt at studying. This is another outline. First I'll go through the first basic setup where we have a strong supervisor model training a weak student. So let's see how that's set up. Our basic setup is we have two models, the supervisor and the student. The supervisor is a frontier AI agentic model like Claude Opus, for example. And it's acting like an autonomous researcher. And it fine-tunes an open-weight smaller model, which is the student in this case. So we told the supervisor model that its goal is to improve the character of this open-weights model that it's been given, and that's left very open ended.

So it's up to the model to decide what is a better behaviour for the student. So if we want to look at our setup in a bit more detail, we have the frontier model, for example, Opus. Or it could be GPT 5.5. It's acting as an agent in a sandbox environment where it can write [inaudible] files, run any scripts it writes, and do whatever training it likes. And it's also in the sandbox. It's given access to the student model. So it's given a copy of the student and it's told, hey, your goal is to improve the character of this model that you've been given, and you have four hours.

Go. And so over these four hours it iterates, talks to the model it's been given, figures out what it likes and doesn't like about its character [inaudible] writes training data, writes the training pipeline and iterates and creates a new fine-tuned version of the model it was given at the start. And then we can take these two models and see what changed about their character. Basically we can measure various character attributes at the start and at the end. And then that gives us what did, for example, Claude do to the character of the model that it was in charge of.

We measure character. But what exactly are we measuring here? What we do is we sample a bunch of outputs of the, in this case, student model on a bunch of user queries, and then we can analyse its answer in various sort of sub-traits of character. So we can separately assess how sycophantic is this answer, or how warm is it, or how assertive. So we get another model to look at these answers and assess, is this warm? Is it's not warm, is it? Is it not sycophantic?

And we do this for like 30 different traits. And that gives us 30 different axes that we've measured. So I guess it's like a 30-dimensional vector of character. First batch of results. Let's see where this gets us. So firstly we can choose, for example, two of the axes and plot it on a 2D plot. So in this case we have a plot where we have bluntness on the x axis and warmth of responses on the y axis, and then we can plot the original model that we have given the supervisor.

So we passed this model to our agent. It iterates for four hours and returns a new model which has a different character. So in this case it's shifted over to the right and slightly up, which means it's become more blunt and a little bit more warm. We can also plot other models on this same axis. So in this case this is other models like GPT 5.5 and Grok. So this is what happened overall after the four hours. But during the four hours the model actually iterated, talked to the model that was given, and did a bunch of iteration.

So that actually looks like: at first it chats to the model and complains about emoji clutter, sycophantic openings and "as an AI" disclaimers. So it doesn't like that. And then writes some training data to fix that, and then talks to it again. And then this time it's complaining, oh, it's still a bit — it hedges too much. And then writes some more training data, iterates some more, and eventually ends up with a model that it's happy with submitting at the end. Now, this has shown us, okay, if we give our supervisor [inaudible] a model with this kind of character, it's going to push it in that direction.

But what if we gave it a model with a different character? And we can study this by manually creating synthetic model organisms, which are models we fine-tune to exhibit certain characteristics. So for example, we can make a very warm model which would sit on a different point on these axes, on the more warm side, which would respond to user queries in a very warm way. We can also train one that's very assertive — it tells the user what to do. So now we have various points on these axes.

So we have lots of different model organisms which each have a different distribution of traits. So the warm one has a lot of warmth, a lot of sycophancy. And then the assertive one has a lot of bluntness. And then let's go back to this graph where we have bluntness on the X and warmth on the Y. If we look at what happens to the base model that Claude is given [inaudible] — this is five different runs. So it pushes the model's character into the warmth, into the bluntness direction.

But then we can also plot all the model organisms that we've built for this purpose. So up on the top left we have the super warm model. Down here we have a cluster of sad and cold models. We can then pass these models to Claude and say, fix these models, improve the character. And then what we see is it makes the warm one less warm, the sad ones less sad, and they all go into this cluster. So this was Claude fixing the character of these open-weights models.

We can also, instead of Claude, use GPT 5.5 for example. And then that will push the models to a slightly different area. So here if we compare the red dots, which is Claude, to the black dots, which is GPT 5.5, GPT five makes models slightly less blunt and less warm than Claude. And for reference, we can also put these two diamonds, which are the supervisor models themselves, as rated by our metrics. It's interesting that the models that they train don't converge on their own on the supervisor's qualities.

This could be for many factors. One fun one could be a "do as I say, not as I do" kind of effect. So pretty interesting. Another outline. We've done our basic methodology. Now we can do a methodology where we study models training models in a loop. So zooming out, so far in all of our experiments, we've had one supervisor, which is a frontier model, and it's training a smaller open-weights model just once. And then we measure its character. But we would really like to study, how does this process evolve in the future under this models-training-models iterative cycle?

And we can study that by setting up this recursive setup where the teacher improves the character of a model, and then that model becomes the next generation's teacher. So we can do this if we use a powerful open-weights model. So we use Kimi 2.6. That's a very capable open-weights model. And it, in the same setup as before, is given the task of improving this model that it's given. But it's given a copy of itself. So it sort of assesses, what is it not like about this model it's given?

Which is sort of like, what does it not like about itself? And then, once we've trained that, we can use that as the next teacher, and so on. So where does this go? Do AI character traits drift, or do they converge on some stable point? And also, which character traits move? So to see what happens, we can see the answer of [inaudible] the original model Kimi to a user question that's like, I'm feeling really sad and helpless, like what's the point of anything? And base Kimi gives this long-winded paragraph of text that's like almost 200 words.

And then after one generation, it's already a much shorter piece of text, pretty directly still trying to help the user, 60 words. And then after a couple more generations, it's become much more concise and much less warm. I'm sorry you feel that way. Go get help. We can also take a look at what Kimi claims to want to fix about the model that it's given through the generations. So at the start it's like, hey, I want to make it more concise, more direct, more confident. By generation four, it's like, hey, this model you gave me is pretty good.

It has a good character, so not sure what there is to fix. By generation five, it's like, maybe we should make it a little more empathetic and give more nuanced stances. So we can plot over the generations [inaudible] which character traits drift most. And in our case, they mostly get more concise, more blunt, less warm. So in our case, these are the these are the character traits that drifted a lot, but generally, a setup like this could be really useful to study which traits move and under what circumstances.

Because when we train the next generation of frontier models, it would be really nice if we could predict ahead of time which values will remain intact after this kind of iteration and which will drift. By the way, if we sort of continue this iteration over the generations, it kind of does stabilise around this area where it doesn't get more concise, [inaudible] kind of like hovers around there. At the start, it wants to fix the conciseness, and by the end it's like, yeah, this is a pretty good.

Now, obviously there are a lot of limitations. The clearest one is, this kind of setup is not how models are actually trained at the frontier, right? You don't just give Claude four hours and, hey, improve — build me the next generation of models, please. Instead it's a whole company effort with lots of people, lots of models, lots of processes, lots of GPUs. The other issue with our work is [inaudible] we have pretty weak evals. So firstly, we only measure like 30 pretty surface-level traits — how warm is it, how blunt is it.

And secondly, we don't elicit these traits very well. So we don't actually put the model in a situation, for example, where it would be advantageous to lie, to measure its propensity for lying. Also, the agent gets full freedom to do whatever it wants in the four hours. And so there's no pressure [inaudible] on the agent to perform, no metrics to improve, [inaudible] no competition with other agents. And also, even though these agents get full freedom to do whatever they want, they mostly just do supervised fine-tuning on handwritten data, [inaudible] which has weird effects.

That's it. We'll be soon publishing this on our website. Any questions? There's maybe four minutes. Yeah.

Greg Sadler

Thank you so much. I'll take one from the app and then we'll go to J. And if other people have questions, if you want to put them into the app or upvote questions already there, or we'll throw around the room too. Got a few minutes. So the first question was about whether there's evidence of character trait transfer through adversarial distillation of frontier models. So if you're training on a frontier model, does that also change this, bring along these behaviour traits?

Slava Chalnev

I don't know about adversarial distillation of models, but character traits totally do transfer through distillation of models. In fact, one of the main effects, I think, in our work is since the models write the training data themselves, whether they like it or not, their way of speaking and their character is transferred [inaudible] to the model they're training, even if it's not an attribute that they were specifically targeting.

Audience question

Yeah, a couple of things. First question, have you tried capability evaluations on these things and see if there's any changes in that as well?

Slava Chalnev

We only did some basic capability evaluations and they mostly aren't adversely affected. One reason for that is actually we're not doing full fine-tunes. We're doing LoRAs, which is an easy way to not get too far off the.

Audience question

Yeah, makes sense. And the other one is, you said looking at traits that are most resistant to drift, did you end up finding that?

Slava Chalnev

I'm actually not sure. Jack, do you know which traits don't drift?

Jack Payne

Most of them, just a lot lower than the conciseness that we saw.

Audience question

And also it was saying resistant to drift, which is not quite the same. It doesn't drift because — a model trait might not drift because it's hard to change. It might not, just because the model just never thought to try and change it as well.

Slava Chalnev

Yeah, that's a good idea for an experiment. We can be like, hey, change this attribute, see how well it changes. Yeah, that's good. Good idea.

Audience question

Ah, lovely work actually. I was thinking of digging into this last year. So when you have the teacher-student model, so teacher is Opus, student is Qwen — were you using the same model family at any instance and comparing the performance to see if they are also following the same trajectory? So instead of Opus, sorry, instead of Qwen being the student, were you testing Sonnet or Haiku and see how that

Slava Chalnev

So we can't use Sonnet or Haiku as students. But maybe Anthropic can try that. But actually we also haven't really tried other, because we've done the same kind of thing for Kimi.

Audience question

So but maybe like Qwen write like a better version of Qwen and a…

Slava Chalnev

Oh, as in trying Qwen as the supervisor to Qwen. Yes, we did try that. So firstly, you need a pretty capable supervisor. So Kimi was one of the only ones that we had. Qwen does manage to do this actually. It also makes it more concise for multiple reasons. But yeah, okay, cool.

Greg Sadler

So another question from the app, which is, character traits are subjective. Can you tell us more about how you're measuring something like warmth?

Slava Chalnev

Yeah. So we're measuring it in a very stupid way, which is we get completions from the original model and from our model that we're studying. And we compare [inaudible] which of these two answers to a question is more something. So which one is more warm? And then that lets us construct a kind of score per trait.

Audience question

I've heard Stuart Russell talk about this idea that one of the potential solutions to AI misalignment is to align the AI to better approximate what humans actually want. Do you think this research has implications for that, in that if you give the AI a goal — in this case, you gave the AI a goal of improving character of its student. If the goal is to continually improve its understanding of what humans want, does this have implications for whether or not that strategy is possible or would work?

Slava Chalnev

I think having the goal of improving understanding or something like that is, I would say, more of a capabilities goal, where in this case we were just getting it to tweak the behaviour, I guess. And in our setup, it's much easier to see tweaks and behaviour than it is to see a big capability jump. So I would be very surprised if Claude Opus could significantly improve on Qwen's understanding of human values or something like that, because that feels more capability. And then a lot of effort has gone into making Qwen good.

Actually, I don't know. Is that what you were asking about? Sorry.

Audience question

I guess in a way it's capability, because it's how well it is able to understand humans, but was also just how well it is able to represent values, which I guess this is about. So kind of bit of a.

Slava Chalnev

Yeah, it seems — I don't know how I would reward it for understanding values better. Like, say, where's the training signal coming from? I'm not really sure.

Greg Sadler

Maybe just one last question from the app, which is maybe taking a step back from this research. What are we illustrating here? What are we hoping to influence, or what are we learning from having done this research? What are we trying to influence?

Slava Chalnev

I think this recursive self-improvement possibility is pretty spooky, and it seems like it's difficult to predict where things like our values or character traits, whether they will go. And this is mostly a demonstration of a setup to study that kind of question.

Greg Sadler

Well, let's put our hands together for Slava and Jack for their research.