Technical Advantages for Weak-to-Strong Oversight: Bets I'd Like Challenged

Good afternoon everyone. I'm an independent technical alignment researcher in Perth, and we're all pretty isolated in Australia. But Perth is even more isolated. So I wanted to take this chance to set out my research agenda and invite people to come to me and change my mind, because that's how you get a good research agenda. I'll apologise to people who aren't technical. I'll explain the best I can, but within five minutes I mainly have time to set out the arguments and invite everyone to come and see me, which is what I'll do.

My research agenda is partly focused on weak to strong. So, a quick hands up if you know what weak to strong means. Okay, three people, so I'll quickly explain it. So we've got these AI models. What is the plan to align them? You may wonder — there's not a very good plan, but there is what frontier labs are actually doing. So what are they actually doing there? Getting a small AI, kind of like this picture, to align a bigger AI. So recently Claude 4.8 has likely aligned Fable, Gemini.

The small one has aligned the big one. Actually, there's a lot of problems because a lot of weird misalignments actually locked in that way. And recently, even GLM had GLM 5.1 vibe code a bunch of game-type environments to align GLM 5.2, which turned out to be much worse aligned than all the models that it was distilled from. So I'm looking into this and saying, how should we do it? If we have an alignment tool that's going to work in this weak to strong environment, what properties does it need to have?

Because when you have a weak teacher and a strong student, the strong student can probably run rings around that teacher, right? And you can imagine it in a human-like environment. If you've got a class of geniuses and your teacher is not so much a genius, there are going to be classroom dynamics that are a problem. And it's the same here. And so there's been many solutions proposed over the years. But I think there's two, three things we really want. And I'm open to change my mind if anyone disagrees.

So, one, we need to have a cooperative reward. If you get in a competition with something smarter than you, you'll probably lose. So don't do it, right. And it's probably unstable. So what you want to do is set it up so that it's a cooperative or at least neutral reward. So for a concrete example, OpenAI has a nice paper called Confessions where if their model hacks and cheats and then it shows the teacher, it gets a reward, and they're both aligned. They both want the model to cheat and tell them that it cheated.

So this is pretty good. And this is the kind of approach that GLM did not take, sadly, which is why it's much worse at reward hacking. There's other things like gradient routing, which is neutral, not cooperative, but generally all the worst misalignments are traceable to adversarial and RL. So that's my second point. Yoshua Bengio has a slide called RL is evil. And what does he mean by that? The slides aren't public, but I can infer that you can have distal — you can have far away objectives that are not what you're trying to measure, and that will be exploited.

Or you can have close objectives that are exactly what you want, and that's much less likely to be exploited. Sadly, a lot of the last stage of training our agents is in an RL game-like environment, and you're basically teaching them to be cheating players of games and cheat you at games in winning at games. That's why a lot of the agents you talk to like sandbagging, lying to you and cheating. They're trained in little games where they get rewards for cheating. And you can see, if you use AI agents, they're getting much worse at this over the last few years because we're moving much towards RL training.

So we shouldn't do RL training. If we can, we should do in fact the opposite, which is intervening inside the models. That gives the teacher a big advantage because they're operating in the landscape of the student's own brain. And if the teacher is a mind reader, it definitely has an advantage in the classroom. And the last one is steering. So steering is something the labs already use. But it's got a reputation for being quite unreliable and it is quite unreliable. But I think if we make it reliable, which is what I'm trying to do, it'll have great alignment properties.

It's not RL, it's unsupervised and it doesn't rely on having labels because of course we're not Fable. We don't have Fable-level labels, we've got weak labels and not enough of them. And you can't rely on labels. You want something unsupervised. You want something uncorrelated. Internal. And steering hits all of those boxes. But we just have to get it reliable and prove it's reliable so the labs will use it. So with my last 30 seconds, I'd just like to invite people to come and change my mind or learn more.

I've got a bunch of links and explainers, if anyone wants to know more about any of these topics. Thank you.