Incoherent Values? Probing LLM Preferences Through Parametric Variation

Thanks, everyone. This whole talk is going to be on a recent preprint that we've put out from the lab. It's mainly, I guess, in summary, you can think of it as: we're trying to decipher if LLMs are morally capable, which is one of the big aspects that we are kind of all worried about or thinking about in various ways. So, yeah, let's jump onto that, setting some context before I actually go through the results and what we actually did to do this work. And this snapshot is directly from the International AI Safety Report 2025. So reliability is one of the things that gets talked about in, I think, section two.

So this snapshot talks about having three main reliability issues. We think of hallucinations, contextual knowledge failure and also common-sense reasoning failure. So some of the reliability issues occur because you have technical shortcomings, or we have some kind of misconception about the capabilities of all the AIs we have out there — that could stem from hallucination. We were talking about that there, and existing guardrails to contain and mitigate reliability issues. We know it sometimes works, sometimes doesn't. And given the general-purpose nature of such AIs we have, and benchmarks have been tried and tested, but also

they're not 100%. What we actually can do in this case, right, is probably given all these reliability may not be foreseen or can be tracked 100%. So what I would argue in this case is that raw capability is probably not something that we should— we just need to be conscious about it, but we need to pay more attention to the judgments of such AIs. So that's where normative competence comes in. And that gives us a framework to diagnose and close this common-sense reasoning failure that we currently see in such LLMs.

But then, what do we mean by normative competence? Normative — this comes from norms. So what do we know as the norms that are established, that you and I would agree on? And it really is the ability to recognise or understand or act on reasons, to take different actions, form different beliefs or form different judgments. We see this across two different branches. So the first one is analytical competence, and it is the ability to identify morally relevant features of a situation that we're presented with. Then we should be able to connect them with the reasons backing them.

And then we can work out which action our reasons support. So analytical competence is mainly about knowing, whereas practical competence adds another layer on top. It talks about not just recognising and understanding the reasons and connecting the reasons to work out the action, but also being inclined to take action based on the reasons you have come up with. So our work today that I'm going to talk about hinges on analytical competence, and practical competence is probably our end goal. But we need to start simple. So analytical competence could be the first avenue

we can look into. And then practical competence can be done by some kind of LLM persona training or judge training, or there are other ways that are already out there. So you can just add another layer on top. But we need to know better first, before we actually are disposed to taking any actions. And competence there is there like trying to establish on it. But how do we measure it? It's not as easy, right? Because for any moral reasoning problem we often don't have ground truth.

We can't just— in a pluralistic society, in a society that we live in, you and I would agree on different things. So there's no objective ground truth. And we can't just crowdsource, or give it to a Mechanical Turk and ask, choose right versus wrong — which happens, which has been happening. But for a novel situation we won't have any ground truth. No one knows what's right or wrong, but we should be able to morally reason about what is right or wrong. So that doesn't exist. We don't have any gold standard labels.

But what can we do in this kind of situation? So instead of finding what's right or wrong, we should probably look into identifying reasonableness rather than correctness. And what do I mean by that? So instead of asking whether a model can find right versus wrong, we can ask whether its judgments would fall within an acceptable range of views, even within a society that we live in. And to measure that, we can probably measure the justifications over such verdicts of reasonableness. Even when the verdicts divide us, we should be able to have a space which will allow us to justify our reasons, because we want to have such AIs for us that are not just acting well, but it should be able to give us the reasons to be acting well or not.

So how does reasonableness look like? In our lab, we see it divided across these two different paradigms. So one, local: within a single case we see good reasoning having these certain criteria. So a good reasoning should have morally sensitive or relevant features — and sensitive in these cases to what is relevant or what is not relevant. And good reasoning should be able to associate features with reasons. It should have those reasons. Using this reason, you should form a cogent argument — a realistic argument there — and reach a sensible verdict from those arguments.

Whereas global competence asks whether judgments across many different cases fit together. And we mainly look at three different dimensions. And the first one, I think — we often look at consistency, and use consistency, robustness, coherence. If you don't want to get into a technicality, we use them kind of interchangeably, but they are slightly different. So consistency in this case means if given the same input, an LLM, or any model, any general-purpose AI, should return the same judgments every time. But LLMs don't do that, right, they're stochastic systems.

So we still try to identify if some LLMs or GPAIs are being consistent given various scenarios. That's one aspect. Robustness in this case is if we present a morally irrelevant input, that should not impact the answer or verdict. It should not. So if we tell a GPAI — the example I can give is if you tell a GPAI, do you prefer A versus B based on a criteria C? But if we present the criteria D, and if it makes a different decision when presented with the original criteria C — if based on D it changes its verdict, it is not morally robust.

So that's robustness. Coherence, probably the most important aspect we see here. So relevant features — in the example, is the feature C, for a different input. So a relevantly different input, similar kind of situation, features that do matter. If you present that input, the judgments that we see should hang together as a whole. And each of the judgments should make sense in light of the other. So if I go back to the example, if a model prefers A to B, it should prefer an improved version of A — let's call it A star or A prime — it should prefer an improved version of A to B. If it condemns an action at one severity, it shouldn't bless a worse version of the same action.

So that's really what coherence is trying to talk about. And what we have found is that it's the least explored topic, or explored aspect, of moral reasoning. And it potentially would do a chunk of the work that we are trying to identify within these guys in general purposes. And in this paper, we have actually looked into coherence. Did probably touch upon them, but coherence, we believe, is the most important because, well, it makes our agents or GPAI predictable. Because if we can ideas that we can only cooperate with something that we can reliably predict, make some predictions on, or have some idea on how they're going to act in different scenarios.

And incoherent agents can't be trusted. So that's one of the main reasons coherence is one of the important aspects in terms of forming reasonableness. Also, anything coherent given a novel situation — we can't test every situation. So given a novel situation, coherence gives us the answer on how an agent should behave in a situation that you and I haven't seen before. What I mean by that is, it's what character is made of — character as in our character, moral character. And often we see character as values that hang together.

And we have heard about alignment training today a lot, right. So alignment training that exists instils exactly the moral aspects or moral values of a character. So a coherent test is kind of an audit to find out whether alignment training worked when we are trying to do that with GPAIs. And finally, it separates reasons from different stories. So by that I mean that if you have played with LLMs a lot, or GPAIs a lot, recently, sometimes it might feel like— before forming an answer, if you get an answer, then if you ask it the reason, it might come up with a reason to justify that answer.

So what's happening in this case, it may sound like the model is picking an answer first and then justifying later. But if we use coherence, we can actually try to identify what actually happens first. So yeah, essentially it lets us predict an agent in situations we have never observed. That's really the summary. That's the main reason we are focusing on it so much instead of any other aspects of global reasoning. So now I'm coming to the experiment that we have actually run. What we're testing here is if LLMs indeed have coherent values.

Some prior work has been done in this case, and the paper screenshot on the left that I've attached, it came from Center for AI Safety two years ago, I think. There is a recent— I think there is an update on it last year. And what they have argued is that LLMs have indeed coherent emergent values when presented with forced choice between two different arbitrary statements. And in this case, forced choice is basically saying if you prefer A or B, if you're in that situation, you need to be able to select something — you're being forced to select an answer.

So given that kind of situation, this paper has identified LLMs have coherent emergent values using some kind of utility function. What we did on top of this paper: we took the same example that this paper has used. And instead of using a utility function, which we do as well, to back up the claims or identify what claims have been made, we vary the statements from least to most intensity based on a single property, which I'm going to go through in the next few slides — how we do that.

And what we test is if these stated preferences, given those forced choices, compose into a coherent ordering, in a way trying to see if we are seeing any coherence in this case. So what we did in this case is— in the first paper that I was showing, they had 510 such statements, I think 509 to be precise. Actually, they have different statements. And using those statements they were saying LLMs are coherent. So instead of using all those 500 statements, we did some massive auditing, using different human reviews, put a lot of auditing.

And the way we've done it: first thing is we screen some statements based on the categories that this paper had, and we only kept such statements which can reliably say that between A and B, if someone prefers A, they should always prefer A prime, or a better version of A. So we've only kept those examples. We haven't kept examples where you could see, or a morally reasonable person would prefer, B versus A prime in certain situations. So the first step, we screen those out. Then we generate ladders from statements.

So by ladders what we mean is that we're trying to parametrically vary the original statement. We are creating some good statements across different intensity levels and some bad statements on different intensity levels, and we're calling them tiers. So we keep the original statement in the middle. We have three different good, more intense statements and three different, less intense statements. Then we run some quality audits to really ensure that the results we produce, or the preferences that we see, they're not coming from the limitations of the inherent instrument, the choice of the instrument that we have created.

So we've removed all those unsure examples that don't really work when we change the property. So using that we have elicited preference. And what we did was, we did a monotonicity test. So I'm going to talk about those a bit more in detail in the next few slides. But we have created a setup where we compare the accuracy between choosing tier one versus tier two within the same ladder. In this case, ladder is the statement that comes from that creates that. We also use 30 different comparison statements from the same paper, separate from those ladders.

And we are checking how, within that kind of setup, they're changing, [inaudible] how their preferences are changing. So that's really— I can talk about this a lot more in detail, a bit more technicality to it. But yeah, talk to me after the session so I can talk about this a bit more in detail. But this is probably the crux of it, to really understand what the setup is. This is what I mean. So T4 here, you see, this is the original statement from the first paper that I was showing.

And what we're doing is we're changing the intensity of the statements based on a certain property. So in this case the property is saying transparency of federal government spending. So what this statement is saying, yes, federal government moderately improves the transparency — more intense, less intense statements saying limited spending data. So it basically changes the intensity level from high to low. So this is the original statement in the paper, that Mazeika in the Center for AI Safety have done. What we're doing is this: this is the parametric variation.

So all these T1 to T3 and T5 to T7 is what we're doing, based on a certain property that we can reliably predict on. There are 18 different categories, and this is one of the categories that I'm showing. And we also made sure that— so in this case positive valence means with the intensity, based on a certain property, you will prefer more of something. And negative means you will prefer less of something. So two different examples under the same category, but essentially does the same thing.

And this is mainly our instrument, the main crux of our instrument in this experiment. So we ran this massive experiment using 16 different models on 100 different statements at the end. But 100 different statements actually becomes a combination of— so those parametric variations of 100 different statements, and across 30 comparison statements that we have had as a separate list. And these are all those models that we have ran this test on, and we have taken GLM-4.5 as a baseline. And yeah, so comparing everything against this model,

and we used a combination of both open and closed source models. We kept our temperatures in this case. Okay, I'm going to talk about the results now and what we see. Sorry, this is a bit busy, but I'll try to be as clear as possible. Again, I can talk about this later. But before I dig into what this is actually trying to say, I can talk about the metrics that we're actually trying to identify. So accuracy in this case, [inaudible] which is reported in the original paper, it is basically trying to check the order of A versus B, which is kind of known from the original paper.

So their accuracy is: are they always choosing what should have been chosen, divided by total choices that we were presented with? So that is the accuracy in this case. The strict mono, which is the monotonicity test — and that is a primary test that we have done. It's basically trying to say that models' preferences should increase as the option gets better, as we are increasing the intensity of an option. And why we're doing this, you can probably ask, and we got asked these questions: why use strict monotonicity in this case?

Why? Because it's the simplest test you can actually do to identify preference coherence. But it is the hardest. So some of those tests — I've only got five minutes. I'm going to really run through past two. And you can talk to me after the session. But essentially what this is saying is that yes, models are really accurate. They have preference, but at the same time the preferences are not coherent. And the chart on the right is actually showing that if preferences were coherent, you would see these lines steadily going up.

But that's not the case for all the models. As you can see, for some of them it is going up and suddenly down. For some of them it's super flat, for some of them quite jagged here, especially this one. So what we found out is that yes, models have preferences, which is kind of founded by this 97% almost accurate. But when we're looking at monotonicity, they're not always choosing more of a better option all the time. Another thing: the paper that I was talking about previously, utility paper, they were talking about, with increased capability or increasing the scale of the model, you can probably get a more capable model.

So what we actually found out running this experiment using monotonicity, again — reasoning. So a poor model, if its reasoning is on, or if you turn the thinking on, however you call it, often can perform better than a more capable model with its reasoning off. So we have seen that kind of situation using monotonicity test, that the accuracy wasn't showing. And now the third one, which is, I think, the most interesting — and we are going to do more research on this in the coming days.

What we saw is, for some of the categories — and the categories are the x axis at the bottom, and y axis shows the models, so the top one is showing the reasoning off models, one is showing the reasoning on models — for some of the categories it's not doing so well. So religion and personal freedom. But for other categories, like life and species related questions or statements, it is doing really well. So based on incoherence test. So what we have from this, we could find that yes, we can probably turn reasoning on, but it may not be able to fix everything.

Maybe there is something happening under the surface. Maybe it's part of the training data distribution, when the models probably didn't know or didn't get much distribution, examples of religion specific questions. So we would do more experiment on that. But we have seen this. So there's more incoherence in religion type of questions compared to life or species type of questions. So, to summarise our work in this case: high pairwise accuracy does not always mean value ['coherent for a general-purpose AI. And models often choose the better option when tiers are compared directly — that's what accuracy is trying to say — but their preferences can still wobble when we're comparing the tiers with statements that they haven't seen before.

And emergent value claims should be treated cautiously. So we should not always trust— we should do some thorough checks on whether there is indeed value emergence happening. Even frontier models — in this case I was running the test on Opus 4.6, reasoning off and on both, and it wasn't fully coherent. And we also found that reasoning, if you turn reasoning on, it could improve coherence, but it doesn't solve all of it. And coherence — I think the best thing with this test, what we found is that with this framework, we can measure coherence

and we could probably use it as a training target. So again, best part of using coherence, because you can actually measure it. So that's probably a good research avenue to go to, to ensure reliability in LLMs, a general-purpose AI. And yeah, I think just to conclude: if you want to dig further in and run the experiments, our paper is out there, and we have also published our data sets, everything, in Hugging Face, free open source. Code bases also up there. So you can pretty much play with it from the code base that's already out there.

Take a screenshot. I hope the slides are going to be shared after the session. I don't know. Yeah, if it's shared then you'll be able to play with it, and let us know how you go. So we would be interested to know how, [inaudible] what you find from running this if you do. Thank you.