AI safety should prioritise democracy

I have modified this talk heavily, even as recently as an hour ago, so this is going to be great. Hopefully, if anyone saw Luke's presentation on democracy and deliberative processes earlier in the day, he communicated a lot of actual information and content. Hopefully that laid the groundwork, because I will have less information and more, hopefully, provocation — or just get across a few ideas that might raise some questions. Now, that was me two years ago. Actually, this photograph was taken right at the same time as the previous AI Safety Forum.

And I have a few more wrinkles since then, but mainly I've become more jaded since then as well. I was an academic philosopher back in 2024, which is already a pretty jaded place to start. But now I am working as a consultant, which I use to fund my independent research into this nexus of AI and democracy, because I think that's a super important thing to focus on. It wasn't the focus of my academic work. I don't have a background in political philosophy, so what I'm going to be talking about today is a bunch of ideas that I've arrived at from a different sort of background, but hopefully that makes them more interesting

because they haven't come through the usual channels. This is where my talk sits, according to the report thing, of course. Due diligence. Just a quick show of hands. How many people here are familiar with the Gradual Disempowerment paper, or the idea that that's communicating? Yeah. Okay, great. So I won't bother summarising it, except to say that I thought this was a really great paper because it took an ultra-realist, somewhat cynical, world-weary or even instrumentalist view of democracy, geopolitics, strategy, things like that, with reference to AI.

So the authors of the paper basically say that as humans become less important to institutions and to governments, humans are therefore less valuable. And so governments have less incentive to care for humans, governments and also non-state actors have less incentive to care about humans. And in the starkest kind of analysis of how history has worked, whenever citizens don't have much value, they tend to get, well... It's not great for the ordinary people when they're not valued by the government. So I was impressed with this kind of analysis, and I ended up sort of arriving at some similar conclusions, again, starting from very different starting points from the authors of this paper.

And so today, I just want to try and communicate one sort of central idea, which will hopefully make you more concerned — if anything, more worried, even more so than you would be after hearing the excellent presentation by Luke earlier today. So I would boil it down to this, that the only reason that any cooperative norms exist in our wonderful democracy of Australia, or indeed any other place where cooperative norms persist, is because of the most important event in human history, which was, interestingly, also the first event of human history.

You probably want to know what that is. Before I get to that, this is what it was like before the first event in human history. It was a dominance hierarchy. So prior to the evolution of humans, every other mammalian species and certainly every other primate species lived in this kind of dominance hierarchy setup, where you have a dominator at the top and a kind of tyrannical alpha male who is stronger than any other individual subordinate. And in theory, if these subordinates could get together and cooperate, they could take down the tyrant, but they lack a way to solve that coordination problem.

Which brings me to the advent of language, which was the most important event in human history because it inaugurated human history. Once our pre-human ancestors evolve - by the way, I'm smushing together about 50 different things from anthropology and evolutionary biology here - but, you know, it's a 13-minute talk. By the way, I want to cover all of human history and some of the future in these 13 minutes. So yeah, I'll be concise. So this is a reverse dominance hierarchy. This is unique to humans, where you have these subordinates who are able to coordinate, and can then extract promises or more cooperative situations from the dominator.

They have leverage. They have bargaining power. And that changes things utterly. Suddenly you have cooperation as the equilibrium rather than some weird glitch thing that might happen occasionally. So that's very important, and I'm arguing that, roughly speaking, language is what allowed this coordination to occur. But you may have noticed that even now we have language, but cooperation still doesn't always happen. If you've read a history book, you'll know that it's not all Kumbaya. As a society grows beyond a kind of small tribe or something like that, you get more layers added to the hierarchy.

And now you have a situation that I'm, roughly speaking, calling autocracy, where you have a sort of elite class who are able to coordinate amongst themselves and extract promises or better treatment from the dominator. But of course, underneath them there is a whole layer of peasants or ordinary people who are still exploited. But amazingly, we've had a second really important event in human history. Arguably there are more, but I'm boiling it down to two. And that was the advent of democracy, which again, was not really one event.

This happened over several hundred years, and I'm smushing together lots of different traditions from political philosophy and history, blah, blah, blah. But this was a new set of coordination technologies like mass media and the public square, urbanisation. All of these things that brought people together allowed them to communicate, to form common knowledge, and to coordinate in order to bargain and leverage for better conditions. And this resulted in a more cooperative equilibrium again compared to the autocracy. The concern, of course, is that AI may be a technology that inherently favours less cooperative norms than more cooperative norms.

The worry is that AI may, in fact, be a technology that erodes the people's ability to coordinate with one another, and that can happen in many different ways. But this, by the way — and obviously all these diagrams are generated by Claude — this is meant to illustrate a sort of AI-enabled autocracy, where you have the elites using AI in various ways that not only increase their power, meaning that you would need a larger coalition of underlings to resist them, but also erode the ability of those underlings or subordinates to coordinate with one another.

And again, lots of other researchers have catalogued the ways in which this may happen, and indeed some of the ways in which it is already happening. And people can look to developments in China to see examples of a kind of AI-boosted or AI-enabled autocracy. So that's the worry, and I think that a huge focus for AI safety should be to try and reverse or combat this trend. And so again, if I had to boil it down to sort of one crucial concept to take away here, it's that you have to try and find new ways, or sort of retrofit old ways, of coordinating with one another in order to maintain any kind of leverage or bargaining power, if you happen to be at the bottom of the pyramid.

Now, I know if I was giving this presentation to some people who were at the top of the pyramid, [inaudible] maybe they would have different ideas, because the future actually looks bright for dominators. So if you happen to be someone who wants to be at the top of the pyramid, then there is every reason to exacerbate the trends that are currently happening. And indeed, this is what the situation would look like in the event that we create something like an artificial superintelligence that is actually so powerful, it doesn't matter how much coordination happens underneath it.

But these are the kinds of people who may well benefit from a scenario in which those at the top of the pyramid have reduced the ways in which those at the bottom can control or coerce their behaviour. Another example. Now, all of these people are obviously from an American context, but if people are keen to be bailed up by me over afternoon tea or something like that about the Australian context, there are plenty of ways in which our national security infrastructure and certain spooky big tech companies that provide excellent surveillance technologies are becoming more and more enmeshed.

And I think that is terrible news. And I think that the people at the bottom of the pyramid, or at least in the middle of the pyramid, should probably try and figure out how to oppose that. That just kind of summarises some good ideas for what we should do. We should focus on coordination, etc., etc. I can see that we've got just a couple of minutes left though, and I would love to take any questions or challenges even in the remaining time. There, by the way,

are some crucial resources if anyone wants to look into this more. These are just some of the things that have influenced my research over the last year or so. I will leave it there because I've covered all of human history. Happy to take questions in the remaining minute and a half.

Emily Grundy

Can we do a round of applause, please. I think the most unintentionally comedic presentation I've seen all day. We don't have any questions in the app, but does anyone have a question that they'd like to do live? Probably have time for one.

Audience question

I just saw your last slide there. At the top it said “apart from don't build a new species that could squish us”. So are you recommending that we don't progress to, or find a way to stop progressing towards, ASI?

Jamie Freestone

Well, there are a couple of negatives in that question. I'm saying we shouldn't build an artificial superintelligence because it would squish us. Yes. So ultimately, I can’t see, based on my background in philosophy and evolutionary game theory and all this kind of stuff, I can't see any way that that is a stable equilibrium for humans personally, and I would love to be talked out of that, because it would be great if it wasn't unstable. But yeah, so that's my top-line recommendation: don't do that. Yeah.

Emily Grundy

All right. We might call it there. And in a couple of minutes we have a presentation on why AI policy needs more China expertise. Can we have one more round of applause for Jamie.