A Scientist, A Believer, a Sceptic and a Diplomat: What Should AI Safety Prioritise Now?

Emily Low

On your right, I've got Johanna Weaver, who's executive director and co-founder of the Tech Policy Design Institute, who will be facilitating the conversation today. Next to Johanna, I have Sean Peters, who's the founder and researcher at Lyptus Research. Second from the left, Lizzie O'Shea, who's a founder and chair at Digital Rights Watch. And on the left, Greg Sadler, CEO at Good Ancestors.

Johanna Weaver

Thank you. I'm really excited to have this conversation today. The premise for this panel was that often when we're talking about AI safety conversations, there's a tension that sometimes people acknowledge and sometimes they don't, and which the Minister actually referred to in his opening remarks yesterday, talking about the immediate harms that are being caused by artificial intelligence, and then the longer term or potentially existential risks that may come in the short term (who knows?). What I'm always interested in is that they're often pitched as in competition with each other

and actually I have three people here on this stage who I really respect, and value each of their opinions and expertise. And yet they address these questions in really different ways. And so the premise of this panel was to bring us together to have a conversation around how three rational, deep experts can come to three very different conclusions, not because we want to highlight the divisions between, but actually to help build understandings of those different perspectives so that we can all do our jobs in a way that is respectful of those different perspectives that are being presented.

So we've got, as we say, a scientist. Sean's taking that role. All of these are kind of, a little bit, binary in their descriptions. We've got Greg playing our believer — and believer in this instance we're classifying as someone who has a firm belief that AGI is not that far away, that artificial general intelligence is or may be around the corner. And you can respond to that in a moment, Greg. And then we have Lizzie, who approaches this from a much more focus on immediate harms that are happening and occurring right now.

And we're calling Lizzie the sceptic because she's kind of sceptical of the AGI conversation. So we're going to start this with a grounding question. So for each of you, what is your view on the likelihood of AGI - or even super AGI, or whatever your interpretation of that is - in what timeframe do you feel like that is a likely possibility? So let's start with you, Sean.

Sean Peters

I think the short answer is we genuinely don't know. As you said, there's a ton of uncertainty. Probably the most interesting thing to just dive into is there's a group, the Forecasting Research Institute, that does a bunch of surveys to answer these kinds of questions. And I think they actually just recently, earlier this year, did almost that exact same question, posed to a wide cross-sectional panel of experts, from people working directly in the field, advocates for change, think tank members and economists, etc., etc., etc. And unsurprisingly, there's a very,

very vast range in what they think. And I think that's worth just dwelling on: how thoughtful and reasonable and credible people can come to such a vast set of conclusions. What's also very interesting is how each of these individuals within this group have an enormous degree of uncertainty themselves. Even when you often see them voicing their opinions with a significantly strong conviction, when they represent their views holistically and with nuance, they often are very, very uncertain. I think I remember seeing that 20% of the individuals on this panel said that they did not think AGI — and they had a pretty strict definition of AGI —

that wouldn't occur by 2100.

Johanna Weaver

And what was the definition of AGI that they were using?

Sean Peters

They were using that 90% of human tasks — essentially anything that a human could do — that 90% of them could be done by an AI model, with 90th percentile expertise within human experts, at five times the reduction in cost as it would cost a human, and that that would be commercially available and deployed. So a pretty strict definition. And that I just represented the one side, where 20% thought it wouldn't even occur this century. But if you looked at the typical expert, they thought there was a 25% chance that this would occur before 2040.

Johanna Weaver

Okay, so that's the scientist's view. I loved the very statistically driven nature of it. Lizzie, for you, how do you approach this question?

Lizzie O'Shea

I don't think it's a real thing, but I'm hesitant to sort of answer because I want to contest the utility of the question in the first place. Because I love my job. But what I would say is, what are the consequences for saying yes or no to this? What are the consequences of spending a huge amount of time worrying about whether it's likely to happen or not? We already know that if industry got its way, for example, in Australia, that we would have ten times the capacity of data centres that we currently have, which would be catastrophic for any measure in terms of dealing with climate change.

So we already have an industry that is interested in destroying the planet for the sake of a certain kind of technology. That's an alignment problem, to use the language of the field. That's a values problem. That is a socio-political problem. And I think answering the question about whether AGI is likely to occur does often downplay to a significant degree — and I think can be quite distracting from — understanding the social, political tendencies that are alive in this industry, that are impacting how government works and whether it is responsive to human needs and whether it's responsive to what Australians want in relation to our policy, and also whether

we can hold industry to account and how we do that. So those are the questions I'm much more interested in. Sometimes it's framed as an immediate versus long-term risk, which I think there's some utility in doing. But I think climate change is a long-term risk. And I'm interested in doing something about it now, because we understand it to be a real problem that we have to solve. And this industry, for example, is working against that. So sometimes I feel that short/long-term distinction is a bit unrealistic, or also disguises the nature of the problem.

I'm interested in understanding how we can govern these systems. Who gets to have a say in how this technology develops, and how we can make sure the interests of everyday people are front and centre, rather than a small coterie of largely men who are controlling the future of this industry.

Johanna Weaver

I agree, often looking at this question of AGI isn't necessarily useful. The reason we started with it for this panel was because often the level of belief that someone has in the nearness of AGI impacts which of the risks you're particularly focused on. And that actually came out of a conversation that Greg and I were having. So, Greg, from your perspective, how do you approach this question of AGI and its nearness or farness?

Greg Sadler

So I sort of agree with what Sean was saying. We're not in the business of making predictions ourselves, but we are in the business of looking at aggregate predictions from experts. So I sort of glanced at some forecasts this morning and they were placing currently weak AGI - those doing many cognitive tasks - in 2028. And the stronger versions of AGI in 2033. So these are relatively short time frames., and I think Sean was saying we should be uncertain about this, but this is not some wild thing that's far away any more.

This is a thing that could be soon. And maybe there is room for doubt about this, but it's certainly the kind of thing that we should be planning for. If experts were telling you that there was a 30% chance of a nuclear war in the next ten years, this would warrant serious action. And I think that these predictions are serious enough and based on science enough that they warrant serious policy action.

Johanna Weaver

So three very different perspectives that have just been canvassed there. Now I want to draw out, given those perspectives, what is the number one risk that you're focused on? If you were going to be sitting around the cabinet table - you have one chance to get the cabinet colleagues to put some money behind addressing one particular harm - what would it be for you, Lizzie?

Lizzie O'Shea

That's tricky, right? Because I actually think addressing harm is critically important, because unless you address harm or acknowledge that it exists, you don't have trust in those systems. Australians, as I'm sure people in the room already know, are some of the least trustworthy, least optimistic about AI. They have increased confidence if they understood that regulations were imposed upon it, which is not the environment we're currently in. So what would I ask government to do? I would ask it to prioritise a sustainable AI industry. That's what I came up with as one.

And the reason I put it that way is I think we need to talk about AI not as something that we should all lean into all the time, because the benefits are there and we just need to spend more money, resources, regulatory efforts in getting take-up, because I don't think that's how it's going to work. You will not have quality take-up without trust, and you will not have a society that works for most people if the values of industry are prioritised over good governance, over democracy, over other kind of social communitarian considerations.

So if I was to put it that way, I would say a more sustainable approach to AI. I don't know that that requires certain government money to be thrown in a particular direction, but it means not necessarily adopting all the time without limits, understanding where we need guardrails and where we also need to say no, right, and prioritising where things work well with machines, where they don't. Understanding that is the critical frame with which we engage with this technology rather than wholesale embrace.

Johanna Weaver

I think a lot of our work — including, I see Zoe sitting here at the front, my co-founder — has really focused on this concept of discerning adoption of AI. Not just this automatic assumption that it's always going to be yes. So, Sean, for you, what do you think is the biggest risk that we need to be focused on?

Sean Peters

I feel like I'm going to be a bad panel member here, and that I just earnestly don't think that there's just one risk to focus on. Actually, when the Australian AISI’s mission statement came out, I was delighted with how grand and broad the remit was. I think that's good, because the number and quantity of risks and the degree of thoughtfulness that's going to be required against all of those is vast. Maybe if I was going to pick one thing that I do worry about, though, is that we're not going to be ambitious and early enough against many of those threats and risk spaces.

So I think the Australian AISI budget equates to approximately 7 to 7.5 million AUD per year, which I just think, against that grand and ambitious remit, is egregiously underfunded. And so I guess that's where my concern lies, is that we're not going to be ambitious and early enough against these problems, rather than my concern being any one problem in particular.

Johanna Weaver

I don't think you're being a bad panellist. I think you're being an excellent panellist. The reason why I'm asking this question, though, is it's real for government, right? They have limited resources, limited people, limited money, and they're having to make these decisions about which of the things that they're leaning into. So, Greg, for you, what do you think is the most important thing for a government to be looking at?

Greg Sadler

So I thought Kate Conroy's Slido was spot on, that this power concentration risk is the top risk for Australia to be thinking about. And that requires some pretty serious and muscular action in the short term to make sure that Australia has real relevance and strategic leverage going into this weird AI future that we're walking into. The other thing that was high on her list was the misuse risks of AI. And there are just concrete, common-sense actions that we need to start taking. I think Kimberlee's talk in the opening

was that governance can be slow, and that's right, but we need to build up that muscle. We know that these risks are coming. We know that there are solutions and mitigations we can apply to them. We just need to start applying those mitigations at the speed that the risks are coming. And I think biosecurity is the most obvious example of that, where the science is becoming increasingly clear that the amount of people who could plausibly get uplifted by AI to build bioweapons is growing, and there are free mitigations that we could take now using existing laws, but we're just not doing it.

And we just need to move quickly on this stuff, and actually do stuff and be muscular, agentic.

Johanna Weaver

And so following on that — in terms of there are things we can be doing now that we're not doing — what do you see as a current blind spot in the conversation? Every debate has blind spots, right? It's difficult to identify them. But when you're talking sort of interdisciplinary as we are now, you can often identify other people's blind spots. So, Greg, for you, what do you think, listening to the conversations that have been going on, is one of the biggest blind spots in the current conversations?

Greg Sadler

Sure. So when we were making our contribution to the international AI safety report, one of the key points that we were making is about unknown unknowns — that things are going to happen in a world where AI becomes increasingly capable and abundant, that are very hard to predict. And I think the most clear way to think about that is currently the sort of practical difficulty of using intelligence to do stuff is a sort of safeguard. It's almost like obscurity, or security by obscurity in our society.

And if intelligence is cheap and abundant, I think there are many things that might break that we don't currently expect. I think that's where the unknown unknowns are going to come from. So what if intelligence is abundant, and what does that mean for things we haven't thought about?

Lizzie O'Shea

But what can I offer? I think it's actually the known knowns that are the blind spot, because at the moment we have an industry — there is not cheap and abundant intelligence. It's costly. There are material realities to developing AI capabilities. And at the moment, in places like the United States, this has resulted in extremely undemocratic tendencies. You just need to look at the horrific data centre being built in — now, I'm going to get this state wrong — in Mississippi? No in Tennessee, Memphis?

Johanna Weaver

I was going to say Virginia.

Lizzie O'Shea

No, I'm not talking about Virginia. I'm talking about the one that's built by Elon Musk. The really large one. I'm sorry about the state. Sorry — Colossus. Exactly. We don't need to know the state. We can call it Colossus, right? That is hugely controversial, right? Democratically elected representatives are arguing against this; there's increasingly calls for moratoriums on building data centres - that has arrived in Australia. So there is not abundant intelligence. The problems are here and now. People do not like the idea of their natural resources being diverted to build out technology that they do not trust, that is centralising wealth

in a tiny coterie of people who are making decisions about how this technology will develop. So the big challenge is how do we have a democratic system that is resilient to some of these threats? Because we need to give people a sense that decisions that are being made in a policy setting involve them, that they're not occurring behind closed doors between politicians and industry reps to decide what the policy will be, to decide whether the data centres can go ahead, to decide whether your harm is something that can just be tolerated by society, because this is all happening

in the name of a brighter future where AGI will solve climate change. Right? So I've talked about sustainability a bit, but there are a range of other harms. Because people also see how AI is being used in warfare, for example, they also see how it's being used in their social service delivery and how they are increasingly being excluded from access to welfare services and the like. So these are the real everyday problems, that are known knowns, that are undermining people's trust in democracy and can lead to increasing polarisation, which is particularly potent in a context of wealth inequality where we are

looking at some of the largest companies humanity's ever seen, the most well-resourced, the biggest selling power, and a failure to transmit that to Australians. Australians don't like this tech, right. So we need to find ways to deal with those known knowns before we start, I would argue, dealing with the unknown unknowns. And in part that's because the governance muscle that you were talking about, Greg, I think is developed by addressing those problems. If we ignore those harms and we say, oh, well, we'll deal with them later because they're small, it means we're less capable

of dealing with some of the bigger problems that are coming down the pipeline. So this is a way to work out how to make these systems democratic, how to elevate people who are vulnerable, who are experiencing harm, get that listened to by relevant policymakers, address it, and then incorporate that governance muscle, that civic dynamic, into future questions that are posed by the industry.

Johanna Weaver

So we've had two responses, though I suspect you're going to take a different approach, Sean. What do you see as blind spots?

Sean Peters

It's interesting. I think it's a really well-posed question, because I find myself in kind of violent agreement with both.

Johanna Weaver

That's actually the point of the panel!

Sean Peters

I suppose though, very deliberately, that they are also in contention with each other at the same time. One of the concerns I have is just coming back to the expert panel that I suggested from the Forecasting Institute. I still worry that society hasn't really properly wrestled with the question of what happens if that 25th percentile — what happens if that first quartile actually happens. So if we get this strong definition of AGI by 2040, how transformative and impactful is that to society? I think the changes could be far, far more radical than society still considers.

And that's a short-term blind spot that we need to address.

Johanna Weaver

So I'm conscious of time. So we've spoken there around the areas of disagreement. I don't actually think that those three positions necessarily have to be in tension. So I'm now going to challenge you each with one minute each to put forward a safety measure, a proposal, an initiative that you think at least one other panel member would agree with. So, Sean, why don't we start with you?

Sean Peters

I would be surprised if you disagreed with this. I'll just repeat what I said earlier. It's not a specific initiative beyond the fact that I think to solve any one of these problems will require far, far more investment and far more ambition than we have, and than we're currently putting into the sector. So I, for example, think that at minimum we should be 10x-ing the AISI budget.

Johanna Weaver

I definitely agree with that. Panellists, do you agree with 10x-ing the AISI budget?

Lizzie O'Shea

I actually was going to propose something similar, because I was trying to get you to agree with me. But part of the reason is, for example, the Productivity Commission's report I think was really interesting, because all it ever talked about was deregulation as a way of opening up the AI industry rather than an investment (which is so telling - it's a classic Productivity Commission play). But that was really missing from the conversation. So you go to these consultations, there's all these people who are working in universities all around the country building

robotics in health care, whatever, and they're trying to get enough compute to do whatever they're doing and they can't get it. And the whole conversation is about hyperscalers from the US coming here and setting up shop. And that is an enormously disappointing position to be in in my view, partly because when you talk to everyday people about AI, they're not actually as concerned about things like AI in certain kinds of medical research. They want it governed well, but they understand the potential benefit there and they want to see it explored, as opposed to AI being used in other contexts

like social welfare or high-stakes contexts like military applications and the like. And so I think that is something that we could lean into as an exercise: how can we invest well in some of the ways in which it might improve people's everyday lives? That's something I think we could agree on. I keep harping on about this, but it does require the right governance mechanisms in place, because you can't just throw money at a problem and expect it to be solved. It needs to have forms of accountability and trust.

But there are other cutting-edge fields of science where we've done that work, where we've tried to build in that kind of form of accountability in public control and ownership. That is worth, I think, following those examples.

Johanna Weaver

So I think there's the governance piece there, but there's also the access to public interest compute, which I think is so often lost in the conversation, which we really need. Greg.

Greg Sadler

So putting a zero on the end of the AISI budget seems like a pretty good idea to me. And I think Lizzie and I share a concern about the tech bros and the power of these companies and the erosion of democracy, and maybe the erosion of how Australia has a place in the global order, that could come from AI. We might disagree on the precise policy prescription to address that, but I think the concern is shared. The policy I'd advocate for is interventions on biosecurity. We know how pandemics affect the most vulnerable people in society in really disproportionate ways.

And we just can't afford to live in a world where a large pool of wrongdoers could use AI to cause pandemics. We can make free preventative interventions on that topic now, and we just need to do it. And I think that's a pretty safe bet.

Johanna Weaver

All right. We've got 30 seconds. So I want each of you in one sentence — actually one sentence ( I'm looking at you, Lizzie): If Australia gets one thing right on AI safety in the next decade, I hope it's...

Sean Peters

Being early and ambitious enough.

Johanna Weaver

Great.

Lizzie O'Shea

Focusing on sustainability and critical procurement; discerning adoption. Perfect.

Greg Sadler

Turning up to the fight.

Johanna Weaver

Thank you very much.