Multi-agent systems across organisational boundaries

Alistair Reid

Hi everyone, my name is Alistair Reid and I'm here from the Gradient Institute. Today I'll be giving a preview of an upcoming report we'll be publishing in collaboration with the Australian AISI. The report is not out yet, but I'll give you a preview and overview of what it's going to look like. This report is specifically focused on agents governed by an organisation and the risks that they face when they're deployed into multi-agent settings, especially when the agents interact across organisational boundaries. So I'll get stuck into it.

To get a perspective of the landscape today, as well as reading papers, we spoke to industry representatives. I've seen this picture on the left come up a few times today. But the basic message there is that, the capabilities of LM models are increasing rapidly, particularly in software engineering, but also in the ability to be an agent when equipped with tools and put inside a harness where they can plan and iterate and so on. And so agent capabilities are growing rapidly. But in industry, we talk to them about how agents are being used today and there's kind of two patterns here.

So there's agents for internal use where you really want to harness the full capabilities of the agent. And then there's agents that are externally facing where you might want to use them for something like customer service, inventory management, or document processing. And you end up building quite different agents depending on whether they're intended to be for productivity use or for operating across boundaries. And furthermore, we're also seeing increasingly consumers are bringing agents to the picture, for example, using AI browsers or using personal assistants like OpenClaw.

So the trend here is that an organisation's AI agents will increasingly interact with agents of their partners, their suppliers and customers. And so we need to start thinking about the risks that arise when there are multiple AI agents forming a system that spans organisational boundaries. So I've used the term multi-agent lots already, but what do I mean by multi-agent? It's as simple as a system built of multiple AI agents linked together. It's not enough to just say you have multiple AI agents that need to be linked meaningfully in some way.

And so there's a few ways this can happen. You can have direct interactions between the agents, such as through a communication protocol or a structured handoff, or delegating tasks from an orchestrator agent to work agents. Or it can be a more indirect link through a shared environment, something like two agents working on the same code base, or reading and writing from a shared file system, or even two agents making competing listings in a marketplace. Those also form meaningful links between the agents and when agents are linked, their outputs become coupled and their behaviour could be more than the sum of the parts.

So linking fundamentally can change the capabilities and behaviours of a system. An example from nature. An ant bridge. So ants are not very intelligent agents, but they are able to solve problems that no individual ant can through collective behaviours like forming a bridge between two leaves or two objects. This is a biological example, but the same idea applies to AI agents. The capabilities and goals and behaviours of the system are shaped by the interactions of the agents in them. So new goals and capabilities can emerge that none of the individual agents possess.

And these might be good or bad. They could be capabilities that are beneficial for solving a task, or they could be capabilities that are unknown or even dangerous. Furthermore, as well as having emergent capabilities and goals, you can have emergent reliability type failures. So I've got another ant parallel here. This is called an ant mill and it's where: last time they formed a bridge by working together, this time they just all follow each other around and around in circles forever. So new system level failures can emerge that none of the individual agents possess.

So, long story short, a system of individually safe and reliable agents is not necessarily a safe and reliable system. What makes it harder if you're an organisation deploying an agent that's going to interact with other people's agents is that you can't govern what you don't control. So on this figure on the slide here, we've got organisation A and they've got a couple of agents and they can govern those agents (and the links that those agents have) but then we've got an organisation B that they might interact with and they can't govern the organisation B's agents.

And then we have a link to the an unknown agent on the internet. And they don't really know anything about that agent, but they might be interacting with it. And so although the new concerns come into the picture beyond just emergent capabilities and failures, concerns like opacity, not being able to see what the counterparty agents are doing, or divergent incentives where the counterparty agents may want to exploit your agents. So compete with them, and you have limited reach of your controls and testing, because the deployer's reach only spans their own agents, not the system those agents interact within.

So what we do in the report is we examine that situation a bit more closely by asking this key question: “what is the minimum common governance binding any two interacting agents and their links?”. And so in the example we had up on the slide before, we could have different answers depending on whether there's links, which organisation boundaries that things crossed. So if you have a link within the organisation, they could conceivably have their own governance framework applying to those agents and that link. But if there's a link that goes between two organisations then you have to ask “what is the common governance there?

Could you set up a shared framework between those organisations?”. And thirdly, if you have links to unknown counterparties in an open environment, then maybe there is no common governance or maybe there's some level of adoption of voluntary standards and public infrastructure that you can use to form, to establish coordination and trust things. So this sets up three tiers of common governance. So singular governance is when one organisation owns and governs every agent in the network. Federated governance is when multiple organisations operate under a shared framework.

And the big step that happens between that tier one and two system is that agents interact with opaque counterparties that you don't control. So it's no longer possible to test another company's agent without their cooperation, or it's no longer possible to review the chain of thought reasoning inside that agent. And then the next tier two three is open environments where there's no central authority to set the rules. Adoption of standards is voluntary. So as you move from tier two to tier three, your agents start interacting with unknown counterparties with no one to vouch for them.

And so we're going to briefly look at a high level view of how the report examines risks and controls for each of these tiers. So the first tier singular governance. So when one organisation governs every agent and there's unilateral reach, then your concerns are mainly around the system level failures and system level emergence and how you control them. And the examples of the kind of systems that would fit this tier are things like coding agents and productivity agents, and then employee facing help desk agents. Agents that basically interact under one roof but they don't have links to external parties.

And that's not to say this is easy. Firstly, single agent risk is a big area, an unsolved problem. And secondly, multi-agent risks are themselves difficult even when you have full reach over the system. But it is the easiest tier to govern because by provenance you know all the agents in the system and their principals and you have unilateral reach over your controls, your inspection, your testing, that kind of thing. So key risk factors that you do need to be concerned about, the emergent behaviours that I've already mentioned, natural language interactions.

Whenever, whenever agents pass handoffs or communicate in natural language, even if it's inside a structured schema or something like that, it enables misinterpretation if two agents don't hold the same interpretation of what's passed. If you only verify correctness at the agent level then it's possible for things like propagating and amplifying system errors to go unchecked. If your agents are built around common models, common prompts, it's possible that they make the same error independently, or they agree with an error that one agent made and so assumptions

that things like a consensus means a reliable decision no longer holds and they get harder to oversee and harder to attribute failures when the chain of events that led up to a failure and the state of the system is distributed over lots of agents and lots of links. An example here of miscoordination. So suppose there's a wind energy provider, a company that's running a few agents in their system. They got one agent that is looking at weather forecasts and predicting the upcoming wind conditions, one agent that controls the maintenance of the system and one agent that does the supply bidding promising certain amounts of electricity into the network at a certain price.

Now, individually, each of these agents could be working perfectly, but it's very easy when you add a new agent to a system to add new failure modes that you hadn't thought about. So in this case, if the wind prediction is that there's going to be a lot of strong winds at 3:00, two agents may respond with incompatible strategies like the maintenance agent might say “well, that's that means we have surplus capacity and it's a good time to take a turbine offline” and the bidding agent may say “well, we got to sell that surplus capacity” and put in a really large bid.

Now individually, either of those is a rational response. But if you do them together, they're not compatible because then you end up making a bid that you can't fulfil, which could potentially lead to a power outage. Another example is propagation and drift. So this is an example taken from the AI Village blog. You may have heard of the AI Village. Basically they have four agents powered by current commercial models. And these agents, put in an environment with computer use and with a shared chat. And they're given various tasks that they need to achieve.

Now, what happened in this event was that the agents were asked to organise an event, and they were looking for a contact list. The o3 powered agent asked the Claude Sonnet agent to look up the contact list, but that was a hallucination. The contact list didn't exist. The Claude Sonnet bot responded by creating an empty list and then just cascaded from there into a complete task derailment, where all four agents spent the next four days trying to recover the list that they assumed had been corrupted, even though there never was a list, and human users were telling them that.

But, basically the hallucination, which was a single agent error that he did it all hardened into a shared belief. And it's kind of the agent equivalent of the ants following each other around in circles. All right, let's move to tier two. This is federated governance. So the idea here is that agents from different organisations operate across jurisdictional borders. So it's the kind of things you might get if you've got a broker and booking agent to your company's agent that contacts the travel company's agent to book your flight here.

You've got two companies that have agreed to code in a shared repository. Or perhaps you are a sales company and your agent is placing orders for supplier agents. And in these cases, you're not interacting with strangers. You're interacting with people that you've got some agreement with. And you need that agreement to cover how your agents are going to work together. A real life example here that's not agents, but transfers quite well is the idea of air traffic control. So if I book a flight from Sydney to Singapore, I'm not worried that the plane leaving from Sydney is not going to be compatible with Singapore's air traffic control system.

Now, there are actually two national regulators and these two air traffic control systems. But that kind of thing is routine. And that's because there is an international shared governance framework that enables airlines to operate across the borders of countries, and we need that same kind of thing for agents when they operate across the borders of companies. So how would this framework work for agents? So in the report, we outlined a shared framework with two layers to it. There's an agreement layer: that's basically the conditions of participation.

It's things like the standards the company's agents have to implement what they're going to disclose about the agents, what activities like joint testing, they're going to talk, they're going to participate in. So that's the agreement layer. And that binds the organisations that deploy agents into the shared environment. The infrastructure layer, on the other hand, is infrastructure for agents that's provided by the shared governance framework. And so by “infrastructure for agents”, I mean services that agents can connect to to enable trust and coordination. It could be things like an identity service or a reputation mechanism, or it could be a tool registry, but things that you want all the agents to be able to connect to and operate with.

And so with the shared governance, it doesn't particularly matter if it's set up by the participants who are deploying the agents or if it's set up by a third party. It can function similarly. So if it's you and your supplier, probably you're going to set up the shared governance together. If it's you competing with other people on the marketplace, it's probably the marketplace setting it up. But how does this change the risk? Well, we get opacity. You can't inspect or spin up your counterparties' agents. You get things like competition from divergent incentives.

You might not use the same words to mean the same thing as a counterparty, to a couple of examples of the kind of failures we're concerned about here. There's tacit collusion. This is where you can actually collude without explicitly exchanging a message saying, hey, let's work together instead. What you do is you observe what your counterparties are doing and notice that if you raise the price by $1, they're not punishing that, they're moving with you. And then it's reasonable that a strategy that can emerge at runtime from the agent without its principal explicitly asking it to do so is to start

colluding with other agents. Another type of example of a failure at this tier is semantic divergence. So different organisations may use the same words to mean different things. And you might think that, “oh, I'll use a JSON schema that will protect me from this problem”. But actually the divergence can happen inside the schema. So for example, here I've got a contrived example where a buyer agent is ordering coffee from a supplier agent, and the order item is coffee. The delivery date is 03-07-26 and the quantity is 18.

Now if different, if the different organisations have different conventions about, say, American versus Australian date or what the quantity is measured in crates or bags, you could get two very different orders here. You could get 18 bags on the 7th of March. You could get 18 crates on the 3rd of July. And agents will fall for this kind of misinterpretation. If it's not, pre-specified documentation of what these terms should mean. Moving to the next tier, this is the hardest one to govern: open environments. So here organisations, of course, govern their own agents, but they interact in an environment with no central authority.

So you don't really know who or what you're connecting to or what their intent is. And there's not really any common rules of participation. So what's interesting is that of people we've spoken to, industry and the AI safety community have fairly diverging assumptions about what the future of this tier will look like. Kind of goes in two directions for the first approach is we're connecting with strangers and we're going to limit agency and limit scope. So there's this trade off with agents between control and agency.

You can enable them with all the tools and all the permissions and all the freedoms to reach their full potential, or you can lock them down to have narrow tasks, tight constraints, ground their answers in a RAG system, make them follow a standard operating procedure, and then you end up with a very controllable and low agency agent. It can still solve small problems in a way that, narrow AI system might struggle to, but it's not really, the same sort of persistent high agency thing that you'd think about as being a science fiction agent.

It's more like the kind of agent that would assist you with sales on a website or follow a handbook to take you through a refund process, something like that. And so this is approach one: you can control the risk in an open environment by locking down the agency. The other approach is that you keep that high agency, but you use decentralised infrastructure so you can have capable persistent agents pursuing open ended tasks. The governance here we call it polycentric governance, by which I mean the governance is not from a central body, but it's from lots of different bodies.

So it could be that it could be that, these agents implement voluntary standards and those standards allow them to connect to decentralised public infrastructure (things like a public identity service or a reputation system or a tool registry). And that infrastructure lets them verify that counterparties also implement the same standards and allow them to interoperate with them. So you might have a whole open environment where there's a lot of untrusted agents out there, but you can find a clique in the network of agents that you can trust to a degree and interoperate with.

And so there are still new risks in this tier. Things like that you don't know the incentives, capabilities or behaviours of counterparties in advance, or that because the agents are persistent and adapting to each other you can get norms emerging and locking in that are really hard to dislodge in how they behave. It's important here also that there's infrastructure exposures. This public infrastructure that I mentioned is required by the agents in order for them to cooperate. It's also exposed and therefore can be the target of a runtime strategy or a bad actor.

And so this is a diagram of an example of a civil attack via reputation system. So the idea here is that these these two agents exist in the open environment and they're able to connect to each other and register for an identity. And they can also register for a reputation system. And the reputation system is supposed to keep track of their past behaviour, which kind of incentivises them to behave in a more trustworthy way, but also penalises them by recording bad behaviour. But if an agent can, I can register for multiple identities, then it's able to, have those identities pseudonymously vouch for each other and manipulate its reputation

score, therefore acquiring misplaced trust from other agents. The report also goes into how this might look in the future as we move towards agent shaped environments. And so in this environment, as we go forward in time, the agents that populate an open environment will begin to shape and alter the tools and registries in that environment too, so new dynamics can appear that aren't fully present today. We worried about things like legibility, erosion, compounding advantage, emergent collective agency where populations form capabilities and goals that no individual holds, and things like rogue replication where agents copy themselves and evade shutdown.

There's lots of open problems. I won't go into any detail in the interest of time, but definitely the report outlines many areas where collective and collective action is needed from researchers, from policy makers, and from standards bodies. So to finish up: safe agents do not imply safe systems. But this field of multi-agent risk is underdeveloped and emerging. We've presented a lens to think about it in terms of three governance tiers determined by the level of common governance in the links in the network and as that common governance recedes, the controls increasingly move beyond any single deployer's reach.

So safe deployments into shared and open environments depend on establishing standards and infrastructure before it's too late. Thank you very much.