Beyond the Model Boundary: A Systems Perspective on AI Safety

Good morning. Thank you for being here. I'm from CSIRO. Today we'll be walking through the perspective. It's all based on our synthesising of knowledge works, which is under review at the moment and will be available soon on arXiv. So it's beyond the model boundary. I've been hearing this topic a little bit in the preliminary talks and keynotes and all the other talks, but I would like to provide some aspects — that means what not to miss. That means whenever we talk about the model safety or AI safety, then we mostly think about within the model boundary.

That means how we can improve the bias, how we can reduce the hallucinations, how we can improve the model accuracy, so and so forth. But in reality, this aspect is something different, right? So the interconnected systems — AI is connected to data centres, vendors, APIs, agentic workflows, legacy. It is so many things interconnected. The point that I'm trying to make on this slide is not just building better models. And it is also about understanding what happens when these opaque models become the embedded dependencies in our safety-critical infrastructure.

So carrying on, that means how the deployment ecosystems would shift the things from how we think from the research or the benchmarking or something. So how it goes into the real ecosystem. So it moves from the evaluation metrics, offline evaluation, controlled test conditions, bounded adversaries, full observability. Right. Everything is under our control. But once it goes to the deployments, then we need to think about the continuous operations, legacy. It integrates safety-critical timing and partial observability, cascading failures. Now, what the thing shifting from our legacy critical infrastructure security perspective to the frontier AI perspective.

So moving, shifting from the bounded and deterministic system behaviour to the probabilistic and adaptive. Segmented operational network to the cross-system software dependencies. Component transparency and known supply chains to the opaque retrieval corpora and vendor-controlled black boxes. And then the human decision cycles to the machine-speed executions. So many things are shifting. So the next few slides I'll walk you through what can be the way we [inaudible] we can think about the shift. So the first thing is the accelerator. That's the emerging capacities. There's a traditional attack life cycle.

So it's within the human decisions period, like weeks to months. And now what the AI is doing, or what the frontier AI is doing, is automated — the vulnerability mapping, rapid prototyping, rapid optimisations. It's reduced to hours. So the point that I'm trying to make here is the AI democratisation doesn't just automate the tasks, it creates an asymmetric tempo advantage, where the attackers adapt faster than the institutional approval cycles can respond. Now we know that was a problem, right? Problem is the speed and capabilities. And the next — this slide.

What we are asking is how did failures sneak into our systems. So it can be various ways. Training data, integrity sites, poisoned data, and then the opaque supply chains, and then the shadow AI – where the unsanctioned generative AI tools and undocumented workflows. The point that I'm trying to make here on this slide is, vulnerability not just about the prompt injection. So it's more than that one. So next slide is how the cascade effects can pass from one system to the other systems. The point that I'm trying to make is the shared foundational models and the software coupling create the common-mode exposure, turning the localised AI failures into the multi-sector cascades.

So imagine the situations, the user traditional situation. Energy systems used to be air-gapped and isolated. But what happens with the introduction of AI models and other things is it adds the software coupling in between the systems. For an example, weather forecast data. If it is poisoned, it can easily pass to the energy systems and then through the shared foundation models. Right, or the coupling — it can pass to the other sectors like the telecom sectors. And then the whole cascading effect, so you can see it.

The next thing is why we are losing the controls. The main reason for that one is the speed oversight gap. So how the institutional response capacity usually looks like is more on the SOC — that means Security Operations Centre playbooks. Right. So human validation, regulatory approvals. But what's happened with the AI side is the decision speed. So the point that I'm trying to make here is nominal human authority is useless if intervention windows are exceeded. That's a problem. So, summarising everything — capacity emergence, infiltration pathways, cross-system propagation, response capacity.

It looks like the dynamics ecosystem. And then, concluding the presentation — safer AI requires safer systems. And how are we going to mitigate it? I list out the three points here: bounded deployment, provenance-aware dependencies and AI-specific incident coordination. This concludes my presentation. Thank you.