Pausing the Frontier

Let's talk about a malfunction that poses existential risk. And then I will propose what scale of safeguards and monitoring will be necessary. The International AI Safety Report defines loss of control as scenarios where AI systems operate outside of anyone's control, with no clear path to regaining control. Some experts believe this could lead to human extinction. This should make you interested in what is the likelihood of that. Let's look at the largest survey of AI researchers ever run. AI researchers give human extinction or similarly bad outcomes at least a median 5% chance, a mean 14% chance.

That is a 1 in 20 chance that their work ends humanity, according to the people doing the work. In the same survey, researchers averaged a 25% chance that human-level machine intelligence will arrive over the next decade. So this will plausibly happen soon. According to Toby Ord, an Aussie researcher at the University of Oxford, AI poses a greater risk of extinction than nuclear weapons, plus pandemics, plus climate change, plus every other risk of extinction put together. So what can Australia do about it? We could stop catastrophic AI from being built in Sydney, but it could be built in Melbourne.

What if we stop catastrophic AI being built across Australia? AI can reach across borders. If anyone builds catastrophic AI, then Australians are in danger. This is why we need a global treaty for a conditional pause on frontier AI. The two most important countries to sign are the US and China, because they have the largest data centres and they lead frontier development. But this would still be insufficient, as the risk comes from anywhere that can have enough compute, and a single holdout can build catastrophic AI or threaten to.

The only thing that actually lowers the risk is a verifiable global treaty covering everyone who could build it. You might think this is impossible, but Cold War rivals ran mutual nuclear inspections even at the height of hostilities. And imagine what options open up when national leaders internalise the 1 in 20 chance of extinction and what it means for their friends and families. A verifiable treaty has three parts. A cap on the scale of training runs until there is scientific consensus that larger AI models are safe.

Capping those inputs controls the capability outputs that come out the other end. The most tractable right now is the compute. The second step would be verification on necessary inputs. This could be modelled on how the International Atomic Energy Agency inspects nuclear material. And finally, we need enforcement. We have a toolkit from past arms control agreements and non-proliferation agreements, which include peer pressure, economic sanctions and asset restrictions. That is the model set out in treaty proposals that you can see in the sources, drawing on compute governance literature.

We have many of the building blocks. We need the political will to implement them. Australia is well placed. We don't build frontier models, making us a credible convener. And we've done this before. ICAN, the campaign that drove the nuclear ban treaty and won a Nobel Peace Prize, started in Melbourne. The Australia Group, which we founded and we still chair, harmonises export controls on chemical and biological weapons across 42 countries. We are also a US ally with trade ties to China, and we've just stood up an AI Safety Institute.

Australia also has agency in the AI supply chain in numerous ways. We have a lot more leverage than just building data centres, as shown in the sources and also described in Anton Leicht's recent presentation. Now here are the counter-arguments that I hear most often. First, a pause would let more reckless actors take the lead, and a unilateral pause could do that. Which is why we want a global, verifiable treaty. Next objection is near-term harms matter more than these risks that are years away. I can tell you that a frontier pause would also cut down on AI-driven bio, cyber and disinformation risk.

This would buy time for the valuable work being done by many of the people in this room and across this forum. Final main objection is that extinction risk is speculative. So why would we devote substantial resources towards this? I would say it will remain speculative until the evidence of it is undeniable. And at that point it might be too late. Treaties and inspection bodies take months or even years to set up. We should put them in place now for the scenario where we all agree that we need to pause the frontier.

If you have any objections I didn't cover, there's my email, or you can talk to our global community or talk to anyone here who's wearing the pause pin. Thank you very much.