The Case for Mandatory AI Monitoring & Incident Reporting

Devon Whittle

Thanks, Emily, and thank you all for coming along this morning. It might be a bit of a drier topic than some of the safety concerns that we heard about this morning, but hopefully it will still be of interest. I have 11 minutes that are counting down in front of me, so this will be a whirlwind tour. Feel free to come up to me at the break or in the office hours tomorrow if you want to discuss further. This slide is just who we are at Global Shield.

We have offices in Washington, working to NATO, and in Australia. We work on global catastrophic risk. In Australia, about half of that work is on AI risk policy advocacy. And situating us in the safety report. We are in 3(c) - technical safeguards and monitoring. And the safety report actually has some useful language that informs why this topic is important. I've got some quotes on screen. The first one is around how slow it is that we actually know that harm is occurring. We all see headlines every now and then around people being hurt by AI or AI psychosis or AI cyber attacks.

But it's not systematic. It's not comprehensive, and we know that it is a good line of defence to have that sort of insight into what these companies are doing and how their products are being used. So in three steps, what are we going to be talking about? First, there's the problem that we're trying to solve, and that is that we just don't know what's happening with these models a lot of the time. We won't actually spend any time on this point today, because we only have now nine minutes.

But just know this is the problem we're trying to solve. Second, what's the solution? Mandatory incident monitoring reporting. So I'll step you through what that means and what it could look like in Australia. And third, how do we know this will work? Is this something new? Is this a crazy idea? No. We have examples in sectors in Australia or on policy issues in Australia, but also overseas in relation to AI in particular. So what is incident monitoring and reporting? This is a very high-level overview of how these systems work in existing sectors in Australia.

The first step is an obligation on companies and industry to monitor how their products are being used. Are safety incidents occurring? And then to notify an agency or a regulator when something goes wrong, often with deadlines attached. Then we have this idea that the agency that gets the report needs to dig into it and work out what went wrong. What could be done differently to stop it happening in the future? There are two key ways this investigation function can occur. The first is what's called a no-blame, no liability approach.

So the Australian Transport Safety Bureau is an example of this. Their role is really to find out the cause of the problem was rather than to assign liability. And the reason you might take this approach is to encourage transparency from industry. So if industry knows they're not going to go to court over a report. If they fess up to an issue, that will help them to come forward with issues. Other agencies like the TGA, that's the Therapeutic Goods Administration or the ACCC, they can impose actual liability when things go wrong.

The next step is to act. So as I said, the TGA and ACCC, they can recall products, they can issue advice to consumers around what's gone wrong and how to avoid the issue. And then there's a learning function. So one of the benefits of monitoring and reporting is that at the end of the day, we have a database of what's gone wrong, where it's gone wrong. We can do a systemic assessment about issues across an industry or sector. And as I said, this is pretty common actually in Australia across a whole range of industries and policy issues.

This doesn't even cover the full range of where this is. For example, it doesn't cover financial services. It doesn't have the state-based authorities that have similar functions. But you can see that other big corporations are doing monitoring and reporting. Why not AI? And so you might think, well, Devon, you just said there's a whole bunch of monitoring and reporting already underway on data privacy, on consumer protection. Why do we want to add another duplicative regime just for AI? And the point that I want to make is that at the moment, you have specific regimes on specific issues that can capture an issue when it's caused by an AI system or the use of an AI model.

So, for example, if someone uses an AI model to breach a database and get someone's personal information, the breach itself will be reported to the information commissioner. But the fact that the model was used to cause that breach or is involved in the breach may not be part of what the OAIC is caring about when they're undertaking their response to that issue. Similarly for these other regimes as well. And so that means we're missing the analysis and information around model-level failures. So when the AI model itself is the cause of the problem.

And we're missing the analysis around misuse at scale and the compilation of data around when AI gets misused or malfunctions. That prevents us from being able to look at the systemic risk that we're facing. So a malfunction in an AI model could occur across all of these different sectors. And if an agency is only looking at their siloed issue, they may miss that the problem is with the model itself. I think the most pressing thing about a lot of this is also that these regimes often focus on the SMEs or the corporations

deploying the AI product, or the people being harmed by the AI product. So, for example, critical infrastructure, if you run a power plant and you get hacked by a cyber attack, you have to report it. But if the AI company knows that their models were used for that cyber attack, they're not under any obligation to report it. So the real key thing is the people who are aware that the harm is occurring aren't obliged to report it. I'll be very quick on this slide in the last five minutes, but just to flag that, what we're proposing at Global Shield is not something entirely new to the world.

The EU is sort of the gold standard in AI incident monitoring and reporting. Their regime comes into effect in August of this year. But essentially it covers high-risk AI and it sets certain definitions around what a serious incident is and requires the AI companies to report when their models are used or cause these serious incidents. And I should say, because one of the questions we often get is we can't overregulate; we can't discourage investment. All AI companies have signed up to the EU Code of Practice around its AI Act, including around monitoring and reporting.

So they're all committed to undertake monitoring and reporting under the EU system, except for I think it's Meta, who had some issues around the Code of Practice, but they will still have to comply. And then even in the US, even in California, which is the home of all the big AI companies, there are obligations already in force there, that when we have a real serious catastrophic harm take place, AI companies need to report it. The key difference between these regimes is the level of harm that triggers the reporting requirement.

So in the US systems, it's quite a high bar. It's sort of 100 deaths. Or you know, I think it's hundreds of millions of dollars of damages being caused, whereas the EU is a much broader net that gets captured around the harms. And then even in our region, Vietnam and South Korea, we haven't detailed them on the slide because their systems are less fully developed yet. But their early AI acts both contain obligations on AI companies and deployers to be monitoring their systems for misuse and harm.

So what could this look like in Australia? So the model basically follows the first slide that I went through around what incident monitoring is. We have an obligation to detect, to monitor your AI system, to see when it's being misused or when it malfunctions. We have the reporting obligation, and I split this into voluntary and mandatory, because the focus of the previous slides was around mandatory reporting for the big AI companies. But one of the things we are also thinking about is the value of voluntary reporting options.

When we talk to people interested in AI and average consumers, one of the concerns is a real lack of certainty about who you can talk to. We had the story of someone who went to a GP, and they were forced to use an AI transcription service. Who do you complain to about that? Is it the medical regulator? Is it the OAIC? What's the right avenue to go? So it might be useful to set up some voluntary reporting. We’d just need to work out coordination and resourcing around that.

And then a response capacity coordination investigation. And then hopefully also intervention. So intervention to fix harms when they arise could occur through bespoke powers specific to AI. But in the early days it may take the form of just recognising that an incident has happened and identifying existing powers that could be used to intervene. So, for example, in critical infrastructure, the Department of Home Affairs has some powers to intervene to correct incidents that happen in that space. This reporting system could feed into that to help the minister decide to trigger those powers.

And then a learning function. So again, databases and reporting around when things are going wrong to help the industry improve and help the community improve around AI risk. Very quickly, this is sort of a scalable model approach that we've been thinking about and advocating for. So if we start with something voluntary, just to get the taxonomies in place to make sure the platforms are working. Then moving to mandatory obligations on companies, and then hopefully we get a comprehensive regulator that can actually take action when it's clear there's a systemic risk that needs to be corrected.

I've only got a minute left, so I won't go through these in detail. I think maybe the two things I will say are: won't this deter adoption? As I said before, we already have similar systems overseas that these companies will be complying with. So it's not about adding new regulatory burden on those companies; it's just requiring them to tell Australian regulators when things go wrong. So at the moment under the EU regime, the EU AI office will be finding out about harms and systemic risks that the Australian AISI or DISR may not be finding out about.

So we're not about duplication; we're about making sure Australians are informed when things go wrong as well. And then how can this be enforced? So we already see OpenAI and Anthropic having a presence in Australia. Microsoft obviously has a presence. Meta has a presence. So our market access creates the jurisdictional link that can be used to make these companies comply with these obligations. So that's the argument, in brief. I've got 30s if you want to ask a question, but otherwise very happy to talk over coffee or lunch or at office hours tomorrow.

Thank you.