Australia's AI Safety Institute

Kate Conroy

What an honour it is to be here today with everyone here. I’ve just enjoyed it so much, Meeting people and connecting with friends and colleagues and new people. So it's been a really wonderful day, and it's a real honour to sort of close it off on day one. So I'd also like to end our day with an acknowledgment of the people, part of the Eora Nation as the traditional owners of the land where we're meeting today, and to thank them for their continuing care of our country.

And I pay my respects to their elders, past, present and emerging. So the human fascination with intelligent machines goes back thousands of years, such as Homer's Phaeacians, who were said to possess autonomous ships that needed neither pilot nor steering oar to take passengers exactly where in the world they want to go, avoiding all obstacles along the way. And were tireless and invulnerable until they turned into stone by an angry Poseidon. And humans have been using logic, values and the material world to create artificial computers to help with decisions

for hundreds of years. From the 18th century punch card loom, Mercury channels in World War II, to today’s silicon chips carved with extreme ultraviolet lithography. In 2026, frontier AI capabilities are rapidly evolving, taking on some of the most ambitious scientific and creative work. While safety engineers endeavour to ensure that these capabilities are not misapplied. The UN Secretary-General Antonio Guterres, says, when considering artificial intelligence that the world cannot govern what it cannot understand. And in the UN Preliminary Report of the Independent International Scientific Panel on AI, the authors note that there are still no settled methods, measurement and evidence of AI effect on the economy, jobs and work; the extent of the risk of malicious use of chemical and biological technologies by non-state actors; the impacts on the environment and resources in the global supply chain; and the effectiveness of governance instruments or the effects at an individual and collective level,

such as epistemic erosion, civic participation, and social cohesion. In June 2026, Australia launched the AI Safety Institute as part of the National AI Plan to ensure that Australians can safely benefit from this moment of fast technological change. The Safety Institute is part of the government's ambition to position Australia as a leader in responsible, inclusive and innovative AI development and adoption. I have slides. I should probably do that. That's alright. That one's a little minor one. And indeed, the AI Safety Institute will take an ethical approach aligned with public sector values, AI safety science, priorities from the International Network for Advanced AI Measurement, Evaluation, and Science, as well as data ethics.

The AISI will work across government to support best practice regulation, advise where updates to legislation might be needed, and coordinate timely and consistent action to protect Australians. Our approach means that AISI focuses on risks with scaled impacts across the whole of society, such as ungoverned agentic AI behaviours, risks with the potential to cause existential and catastrophic harm that could cause significant disruption and harm to society (such as AI designed biosecurity threats), as well as helping prevent disproportionate harm to the most vulnerable and marginalised in society,

because it is mindful that significant data and evidence gaps exist to enable our understanding of how certain parts of society are impacted by AI. These data and evidence gaps are particularly relevant to groups who bear disproportionate algorithmic harms and have the least power to seek redress. The AISI seeks to improve the ethical means by which data from low data groups can contribute to knowledge of AI safety risks. And this ensures our work is aligned with public value and supports a whole of government approach to AI safety.

You know, we seek the best information about what advanced AI systems can do and how they affect society. Here are three capabilities the AI Safety Institute bring to the Australian system. You know, first we're going to monitor emerging AI developments and implications for government, including analysing and testing frontier AI models and applications. We test advanced systems with our technical partners, such as ASD and CSIRO to understand how they behave (including how they might fail), or how they might be misused, as well as opportunities they present.

Second, we support regulators and agencies in responding to emerging AI capabilities, risks, harms and trends. To do this, we combine the results of our systems testing with active monitoring of risks and harms. AI systems are constantly evolving as they interact with real world environments. So we'll be scanning the horizon, identify risks early, and build a shared understanding across government before those risks scale. Then we will translate those complex technical developments into clear, evidence based insights so that agencies, regulators and policymakers can act with confidence. We're not a regulator, but instead a conduit to inform the whole of government about AI system capabilities, risks and harms.

And our third role is to shape safe AI development, deployment and international governance in Australia's interests. AI is a global phenomenon with global implications, and we're connecting with Australians and listening to what Australians think about AI risks and harms through events such as this AI Safety Forum. You know, we're forming partnerships with organisations and research partners such as our colleagues at the CSIRO. We work with government colleagues to support best practice in evolving AI governance, including the National AI Centre and the Digital Transformation Agency and the Department of Finance.

We work closely with the International Network for the Advanced AI Measurement, Evaluation and Science, to ensure that global norms are in Australia's interests and are contributing to keeping Australians safe. So our role is to support, not direct, the work of Australian agencies and regulators to ensure that Australia's laws keep pace with technological change and reflecting the government's focus in the National AI Plan on keeping Australians safe. We at the Australian AI Safety Institute, the AI safety as a public good. In a rapid adoption environment the cost of being late to understand AI systems is high.

We are seeing AI systems that can plan, act and interact in increasingly complex ways. We're seeing systems that may behave differently under testing conditions than they do when they're deployed. And we're seeing risks emerge in new and sometimes unpredictable forms. That's why the institute's focus is not just on today's risks, but what is coming next. Working with our policy colleagues in the Department of Industry, Science and Resources and elsewhere in government, we're part of building an anticipatory capability founded in real world data and technical analysis and testing that helps Australia stay ahead of the curve.

And to that end, I'd like to spend my remaining time with you to better understand what you think matters and what we should know to help us on our journey. Thank you. All right. So the first opportunity to interact is we're asking what are your top three of these risks. And everyone will have them ordered differently because I love a randomisation test because I love data. And while we're adding that information to Slido I want to open the conversation. So if you would like to talk to me about one of your choices that you've put into your top three, or another aspect of the risks on the screen here, feel free to raise a hand.

Let's have a conversation.

Audience question

Hi, my name is Rohan and I'm a student at La Trobe University. My honours research is on AI and offensive cyber warfare. So I think weapons and cyber attacks as my top concern. I was recently watching the documentary on ABC where they were just explaining how AI is being used in drones in Ukraine and how it's automated so much so that the drones can automatically detect vehicles and attack them directly. So that was a bit concerning in the way we are giving the power to machines to attack infrastructure.

Thank you.

Kate Conroy

Thank you. Was that a question or just a comment? A comment. Great. Okay. Thank you.

Audience question

Thanks, Kate. I'd love to know a little bit more about how you chose these risks. And it does seem like some of them might be related to one another. So when I look at the results, like if we're looking at them per vote, I feel like there are some categories that may not show up in the top ones, that are related.

Kate Conroy

Yeah. Excellent. So these were actually not picked by us at the AI Safety Institute. They actually come from an MIT risk prioritisation study, which was the Delphi study, which interviewed about 250 (I'm putting in air quotes) international AI experts to find out what they think are the most important risks. And the methodology is available on a paper that I'll provide a link to. Emily Low: Yeah, we’re actually very lucky to have Zan here who worked on that study. So I might pass the mic. So how did you pick the risks?

Alexander Saeri:

Audience question

So this is from a systematic review that we conducted in 2024 and have been trying to keep updated that picked more than 70 papers which have discussed different kinds of risks, and essentially tried to synthesise and aggregate them. But we know that it is not perfect. And in fact, we're working on an update to it.

Kate Conroy

This is really fun. So I was lucky enough to speak at the New Zealand AI Safety Forum on Sunday morning (and shout out to New Zealanders that are working in this space). It was really lovely to speak with them, and I was actually fascinated. Right. Because in the MIT risk prioritisation outcome, online fraud and harms was the number one. And in New Zealand power centralisation was number one (just like in Australia). And number two for New Zealand was dangerous capabilities. And number three was AI misalignment.

So we are very close to New Zealand right. It's so interesting. Right. And so I love this moment because it just gives us this tiny little snapshot just to say, hey, you know, we have our own point of view around here. Even if the classification of risk isn't quite what we want it to be. We can now compare what we think is important to this MIT study. So I appreciate the effort of the team. I know that it's difficult. These research methodologies are challenging. Fabulous. Okay.

Was there another question? Yeah. Toby. Yeah. Hi.

Audience question

Toby Walsh: Good luck in the new post, Kate. I want to talk about a risk that's not on this list yet. And that's the risk to us in particular, as we have AI relationships, AI companions, AI therapists. I really worry what's going to happen to humans and human society.

Kate Conroy

That’s not on the list as far as I can tell? No, correct. Correct. Very good. Yeah. Thank you.

Audience question

Hi, Kate. I have a question on how we might conceptualise some of the more kind of, philosophical risks, like something like scams and fraud. I understand, and I can conceptualise that risk. Something like, the loss of consensus reality I really only have, like, The Matrix to go off. So how is the Institute conceptualising some of the more, I guess, high level risk that you can see might materialise, but a harder to grasp.

Kate Conroy

Yeah, yeah. What a great question. Of course, you ask the philosopher the fun question. Yeah. I mean, I think what's extraordinary about the opportunity here is really to think about, well, what are the taxonomies or the ontologies for risks and harms that affect Australians? Where do we need to understand the circumstances that lead to the harms and how to in fact, we understand the harm itself, right? As you say. Well, what's what's the, KPIs on loss of consensus reality. Like, what's the performance metric on consensual reality?

Let alone losing it? Do we feel like we have consensual, you know, do we think we have it right now? I mean, if you're a postmodernist scholar, you might say that perhaps the, you know, sense of reality has already been fractured and perhaps it never existed. So philosophically, you can go down a very deep rabbit warren here. But I think on behalf of the Australian government, our job is to definitely try to take these more abstract ideas and to concretise them. And I think some of the work in government to look at the DISR AI Ethics Principles, which are abstract, you know, principles are abstract.

And then if you look at the national framework for the assurance of AI in government, the implementation of those principles is specified for government. And that builds on the work by the Gradient Institute and CSIRO a couple of years ago, where they operationalise Australia's AI Ethics for business and industry. At the time, back when narrow, AI was all the rage. And so this operationalisation of principle based approach or theoretical terms that, legible to diverse stakeholders and able to be used by regulators or policymakers or decision makers, in order to formulate what good looks like, I think is really important.

And so a big part of the job, I think. Thank you.

Audience question

My question was you indicated the focus on the international governance for the AI safety issue was to ensure it was in Australia's interests. So I'm just wondering why it's not in the focus shouldn't be on humanities interests that, you know, I would hope that the Australian government would take a position, for example, that if AI was being developed through exploitative labor practices in other parts of the world, that would be something from a global governance point of view. That would be unacceptable to Australia.

Kate Conroy

Yeah, what a great point. Take it. I think the intent there was to indicate that I guess no one around the world is going to take Australia's interests as a priority that isn’t Australian. We have Australian law, we have Australian values, we have an Australian set of stakeholders, we have languages. You know, there's over 250 languages in Cape York that the Pama Language Centre is responsible for trying to, you know, nurture and help, indigenous families teach their children. So there are specific Australian data sets, languages, people, culture, values.

And I think that's the intent of the remit there is for us to work with that international network and of course, participate in an international dialogue about what good looks like. And we have, you know, Australians represented on that UN Scientific Panel, you know, so that's wonderful that we have that CSIRO representation there to speak on behalf of that global conversation. But we really focused on thinking about what do Australians need, what is of a concern where the harms actually occurring, but also what's on the horizon.

And to really think that through fully aware of things like the digital divide or those who are not well represented but need to be thought of in how we go about our work.

[unclear]

Thanks for that Kate.

Audience question

Obviously the AISI has limited resourcing like every other government agency. How are you going about prioritising the risks that you're focusing on? Are you using various criteria like risk, consequence or likelihood or popularity? How are you going about that?

Kate Conroy

Yeah. Well, so one of the things I think is great to establish in our first month is that ethical approach, because it actually gives us something where we have concrete actions in terms of prioritisation. So we're not managing every single risk or every single permutation of harm. And it's more about saying, where do we get it as an ability to have a scaled impact on stakeholders. So let's take the category of, the catastrophic and existential risks of frontier AI agentic capabilities. We have a specialised capability in that.

And so we get to really prioritise leveraging our technical expertise. About half of our team are technical experts in this field. And it really is our job to say how we can work inside government with technical expertise, leveraging the international networks and the domestic expertise, the research and the good intellectual work and the industry level work to actually sort of funnel that into sort of digestible and usable information for government regulators and agencies. So that's one way that we prioritise our work, the other in terms of like where the harms are actually occurring to the most Australians, let's say that frauds and scams piece.

Right. Well, the ACCC is working on frauds and scams. So we're going to make sure we work with the ACCC to support their ways of trying to support consumers, for example. Same with the eSafety Commissioner. We know there's concerns around teen health and chat bots and the relationships between these things. We know that there's Nudify apps. We know that there's concerns around misinformation, disinformation, and these sort of different challenges to it, to our society. But there's also parts of government that are leading that work. So what we do is we say, where is the activation in government that is leading on some of these pieces?

And then how does AISI with our specialist technical capabilities support and, and help scale the impacts of, of intervening on those risks in society. So it's really about working with others and finding out like how do we support government and not take on everything ourselves, because that would be impossible and we would not do a good job of it. So we must, you know, choose the priorities where the government is working in those areas, where there's a scale impacts on those catastrophic and existential risks, and then supporting the vulnerable and the marginalised by finding out what do we need to know about how this stuff might be affecting those who are poorly represented and holding in space for them.

You know, holding a space, even if we don't have good data, noting that we don't have that data. And it's something important to help ensure that if you're working in this space, we're interested in if you've got reports, if you've got data or methodologies, and I've seen some of them today, and I know some of you are really passionate in this area, how can we shine a light on some of what you know, so that we can better manage, you know, cohorts like how is AI affecting the disabled

or First Nations people or rural and regional people in Australia and so forth. And it's really us making sure we are paying attention and listening.

Audience question

Hi, Kate. Look, it's quite exciting to be in a room full of people who are sort of looking at the frontier and worried about what's happening over the next couple of years. When I think about the story of social media and, you know, you talked about the eSafety Commissioner, we spent years experiencing regulatory lag where a harm would emerge, and then it would take quite a long time for regulation and policy to come in behind that and start to solve it. And the eSafety Commission is now something turning the tide but she has a huge job to do because she's starting from so far behind to trying to get ahead.

And when I think about artificial intelligence, I can see a very similar trajectory that's going to emerge. So I'm just interested in your thoughts. You know, there are many people here who deal with governments and civil society institutions. Do you have any advice for us on how do we get people out of the “look at the problem of today” type mindsets to look at the problem and be more anticipative about things like regulation and response?

Kate Conroy

Yeah, getting the narrative so that it can be responded to. Looking ahead, I mean, it's an often a challenge, right? Because sometimes the law only reacts when an incident has occurred that causes a level of harm where the attention, the political attention or whatever is, is on it. So it's a big problem. You know, anticipatory regulation is a big issue. But luckily for us, at least, the AISI has a mission to build up that anticipatory capability. So I guess our challenge and our opportunity is to take the best technical information

about these emerging AI systems and then be able to translate that for different stakeholders and to use the right language, perhaps scenario based language or, you know, translating this stuff so that it is digestible, to those who are in the space of regulatory change and policy, and to try and support them in good information to help their decision making. So, yeah. Thank you. Is the second, Slido opportunity is I really do invite you to, add any report or data, or, you know, something you think we should be

aware of that you've done or someone in your networks has done particularly in this Australian space. Right. So any Australian reports, data methods? I'm to be international as well. I don't mean to say it should only be Australian. But, where do you think the best stuff is? Because it'd be lovely today for us to take home with us Christmas, Christmas present bag of of interesting and good data. And I just wanted to, finish off by showing you the, risks that were listed in order, with the MIT Risk Initiative and study.

And you can see they're frauds and scams. It's like totally at the top, right? Power, centralisation, dangerous capabilities, disinformation and influence, false information. AI misalignments. You can see alignment a little bit further down for the MIT crowd. Loss of human agency over reliance and, unsafe use and AI security vulnerabilities and other guys. So, have a look at that report. It's definitely worth your, attention and potentially critique as they develop. But, but they're doing a lot of good work there. And I just wanted to also finish off with, indicating some great resources that the government has.

If you're looking for great, governance frameworks already. So we have a website, which is lovely, the National AI Centre's Guidance for AI Adoption is terrific. It's very digestible with its six essential practices. And it aligns with ISO 42001, which is fantastic. I gave today, you've got plenty of guidance and really snappy URL. The Digital Transformation Agency has a policy for the responsible use of AI in government, and it also has the AI technical standard. So the technical standard is definitely worth a look. The national framework for the assurance of AI in government and the Commonwealth Ombudsman's automated decision making, Better Practice Guide that has an icon, which is exciting.

Nobody else has an icon update but this is a really great product. It's particularly good. It's got a checklist at the back, and particularly if you think about agentic decision making in government is really or in an organisation, the way decisions get made and what you replace your decision making apparatus with, including agents or automated decisions. It's really important to think through it, the logic of the decision. And that's what I really like about that framework. Anyway. That's enough. I'm keeping you from your bias. So thank you.

So much. It's been lovely.