Responsibility Across the AI Value Chain

Thanks so much. Great to see so many of you here interested in this question. I feel like it's one that doesn't quite get enough attention in law and in associated disciplines. And I think it's a pressing issue. Today I'm going to just be presenting a paper that's currently under review of the European Law Journal, co-authored with Kim Weatherall and Jacqui Zang from Sydney Uni. It's called Coordinating risk management in the AI value chain: fictions, frictions and predictions from the European approach. And weirdly, having the rhyme scheme in the title helped me structure the paper, and finally work out how to arrange the paper.

So why the EU AI Act? As the most sort of complete allocation of legislated responsibilities in the AI context, it's a really useful case study for any effort to implement ex-ante risk management across AI value chains. And in any case, I do try and touch base with the situation in Australia as we go through, so that it's not entirely decontextualised for you guys. So let's get into a hypothetical. Let's imagine that we have an AI system that's used in emergency services dispatch. We'll call it Emergence AI.

The system analyses incoming calls and it assigns an urgency score to emergency calls for ambulances, police and fire services. So this type of system is explicitly regulated under the EU AI Act. It counts as a high-risk AI system, and it has a whole bunch of attendant ex-ante risk management obligations: testing, data governance, documentation, explainability, ongoing monitoring, logging, and so on and so forth. So failure of a system like this implicates both AI safety in the sense that we're interested in it here — it harms people physically, basically it leads to physical, very sort of discrete, clear harms — but it also aligns with concerns about fairness, accountability and transparency.

So maybe we can bridge that — the rift between those two sort of discourses in AI governance, which I think, unfortunately, there is quite a rift there. But so let's imagine this system fails. In a discrete instance, failure looks like an ambulance not being sent to the person who most needs it, being sent somewhere else, and somebody as a result dies unnecessarily. That's the sort of critical acute failure mode here. And at a systems level, with those kinds of fairness, accountability, transparency concerns, as is so common in systems that are trained on historical data, we can imagine a systemic bias so that ambulances are consistently

directed towards wealthy neighbourhoods, towards majority ethnic or identity groups, at the expense of minority groups, poorer groups, poorer neighbourhoods. And I would imagine that's probably already the case in terms of emergency services dispatch or something like ambulances. So good risk management, or the fundamental aim of risk management, is to prevent that sort of thing from happening here. We're trying to avoid discriminatory outcomes, and we're trying to avoid unnecessary physical harms. And that means being able to prevent these outcomes, or at least detect something like the systemic problem when it's happening, work out why it's happening and try and take some sort of remedial action.

That's what all that process-based risk management is for, ultimately. You want to be able to stop it happening in the first place, or if it does happen in some sort of unexpected, emergent way, respond appropriately. So if we imagine an AI system, we imagine Emergence AI as a sort of product, a discrete product, or a single integrated service that's built end to end by one developer, then it makes sense to expect the vendor of that system — they're providing it to an emergency services call centre, let's say — well, it makes perfect sense to say all the risk management is on you.

That's your job. You built it. You work out what's wrong with it. You do the risk management. You fix the problems. Great. So the buck stops in this diagram with vendor A, the provider. And that's in effect the way the EU AI Act imagines high-risk AI systems are working. And in a sense this is also more or less how Australian doctrine works. But it depends on the law. Something like negligence or product liability is going to sort of end up in very much the same place.

The provider of this system will be imagined in that legal scheme of, say, consumer law as the manufacturer of that product, and the responsibility will fall to them. But we know that in fact, systems like this are not built from scratch, end to end. Or not always. And certainly in the past sort of 4 or 5 years, that is the pattern and the trend. Systems are assemblages of pre-built components. And the components each have different capabilities and varying degrees of generality in what they can do.

And they're stitched together in various kinds of architectures with various kinds of tooling and middleware, managing the data flow between the components. So we're not talking about an application built end to end from scratch. And we're also not talking about a linear supply chain, where you have an application that's sitting at the end of a very sort of static sequence of components being passed from one to the other, because components that are used in this system in one way may be being used in utterly different systems — something like speech to text or text analysis or whatever — might be being used in completely different ways in other systems: many thousands or millions of other systems.

So it may not be clear to any of the component providers how their contribution will be used and in what context, and what the risk will be and what the interactions will be. And so something like an obligation of data governance, data governance to what end? Right, for any of them. And if we're talking about the person who's assembling this system, again, how are they performing data governance on the speech-to-text model that's being integrated into the text analysis, medical diagnosis and so on and so forth?

So risks of harm can emerge at any point in this value chain. And then the most complex risks, the most intractable and difficult-to-predict and difficult-to-manage risks, emerge from the interactions between components, or between data and components, or between components and contexts, or between components and people. So it's very, very hard for any actor in this value chain supplying a component or some kind of tool to effectively take responsibility for any risk in the system. So there is some degree of recognition of this problem in the EU AI Act, and most AI regulations as they're emerging, whether they're soft regulation or hard regulation, try to clarify to some extent or allocate or shift responsibility in value chains.

But as you can see, most of the responsibility under the EU AI Act is falling on the application provider. So there is also a sort of an emerging discourse in law and governance. And the ideas that have been pitched so far look more or less like this. I'm not doing it justice — there's a lot of great scholarship out there — but more or less like this. Push responsibility up. Recognising that the providers of especially large general-purpose models tend to be very well resourced and they have greater knowledge and capabilities, and it makes sense for them to do more.

There's this idea that transparency is extremely important, because both app providers need good information from upstream actors, and if you're a general-purpose provider who has some responsibilities, you need to sort of know what's going on downstream in order to fulfil your kinds of obligations. And then thirdly, the theme that emerges — and this was the case as the EU AI Act was being drafted and had very limited at first provisions for value chain management — the discourse there was, oh, well, you need some kind of arrangements for contracts, even just to work out, in a given situation where components are being sort of assembled together by multiple parties, who is the provider of that system?

You need to probably just resolve that through contract. And that is all very important. But I think that the gap here has to do with this idea of collaboration and cooperation. So managing the kind of complexity in this highly simplified hypothetical that I've given involves more than just allocating specific jobs to specific people in this sort of linear way, because many of the tasks can only be achieved through active and ongoing collaboration and exchange of information and tasks back and forth in a very dynamic way.

You can only identify the risks and mitigate them and respond to them by combining knowledge and capabilities and capacities across the value chain constantly, and working out how to combine them in a responsive way. So what do we have? Getting down to brass tacks, this is what the EU — there are other provisions that deal with value chain complexity, but this is the main one. I think it's called a responsibility across the value chain or something. That's the actual title of the provision, Article 25. And it has two main elements.

There's a carve-out for open source stuff, which itself complicates things more than I can manage to discuss right now. But the two sort of key bits of this are: on the one hand, if you modify an existing AI system — and that's defined very narrowly, basically an app with an interface — and what you produce as a result of that, either because it was already high risk and it stays high risk, or because it wasn't, and you make it high risk, [inaudible] you become the provider of the new system.

And whether they like it or not, the provider of the initial system that you modified has to help you with your compliance. So that's a really interesting one, but it only applies really narrowly, essentially to other application providers. The other one is essentially this kind of dictate: sort it out with contracts. The provider is to sort these problems out with contracts with suppliers. So that's kind of what it looks like. That's a visualisation. But what this paper does is sort of poke holes in that, which is — it's a little bit easy pickings to do that.

And we do try and be a bit more charitable. And I'll come to some sort of suggestions and reflections and predictions at the end. I think the fictions, frictions and predictions about the ways in which it might actually sort of muddle together and sort of work in some ways. But let's poke holes. Let's have fun now in poke holes. So when does this work? It works when we have a provider that's building a system from scratch or in a relatively simple and linear supply chain where they're getting bespoke components through negotiated contracts.

I hire a software developer and we negotiate the terms, and I say, I want you to build this for me. Then we have a written agreement, and that language is telling. Originally they don't say a contract. They say a written agreement, which implies a process of agreement formation that's supposed to happen here. But of course, in the world of software, modular software, kind of assemblage, that is not often the case. However providers get access to the components — whether they're getting a Docker file or an API, or they're getting model weights and checkpoints — usually they're pre-built and they're provided on standard form contracts, or what we call contracts of adhesion.

You click the click-wrap ‘I agree’. That's it. You don't negotiate. So if in this situation vendor B's component is provided on standard terms, how precisely is vendor A to secure written agreement to get all that cooperation and so on that's needed in order for vendor A — the high-risk AI system provider — [inaudible] to comply? So in some circumstances the incentives might align for vendor B to do this. And we'll talk about that. I'll come back to it if I have time. But in some cases they’ll be like, ‘no, I don't want you to make a high-risk system out of my product’.

And they might even actually explicitly in their terms say, no, you can't. So we have a problem of misaligned incentives and a lack of bargaining power in that scenario. And what about this scenario? What about the suppliers to suppliers where components are integrated into other components? Is that person a third-party supplier of a component of the finished AI system? It seems very unreasonable to expect an AI provider to chase down everyone in the value chain back to the mining company that is extracting the minerals to produce a chip and enter into a written agreement in case there's some sort of problem with the materials that's causing a downstream problem.

You can't do that. So maybe what GDPR does with privacy is, it says if you're the data controller, you do a contract with your processor and they have to back-to-back it with their sub-processors and so on and so forth. But again, that's a fiction, because the more steps there are in which you have to back-to-back the contracts, the more likely it is that you meet with a contract of adhesion, a click-wrapped contract, a non-negotiable contract somewhere along that chain. So it doesn't work. And there's this really interesting move in the literature about the value chain to sort of make a distinction between a sort of a linear supply chain and a value chain, or a value network, which involves exchange of inputs and outputs and value and contribution at every level, horizontally as well as vertically.

So this is a sort of an imaginary of a horizontal situation, with this Emergence AI. Insofar as there are risks that are being produced that must be managed, those risks are being co-created with other third-party suppliers into a bigger architecture of sociotechnical architecture, if you want, so that on a conceptual level, it makes you start thinking, well, what is an AI system? Where are the boundaries of the system drawn? And it's sort of arbitrary in some sense where you draw them. The other obvious point to make is that in this scenario, vendor A has no contractual relationship with vendor Y, vendor X, or vendor Z. They have a contractual relationship with the emergency services entity, and the emergency services entity has no obligations under the EU AI Act to do anything to coordinate their actions.

And this is sort of the model for multi-agent interaction as well. It's a very similar problem. So in one of the presentations yesterday there was a suggestion that sort of multi-agent interaction is different. I don't think it's qualitatively different. The difference is you just made it come out of that person. It's a minor difference in the exact nature of the complexity. But it's a similar problem of complexity and coordination, in the absence of formal contractual relationships. So enough dunking on the AI Act. I do think that the commitment that Europe has shown — and in the face of a lot of pressure internationally from America — [inaudible] to actually just do anything, is to be admired and is to be supported.

And so how will I support that? Obviously making some suggestions about what they could do, but I want to make some predictions about what might happen and the ways in which we might muddle through to some extent with this. So one thing that might happen is that suppliers of high-risk systems will lean increasingly on deployers and say, ‘you want me to provide you with a high-risk system? I need some undertakings from you that you will go across your suppliers and secure their cooperation for me, and you'll indemnify me for any losses where I'm not able to get that cooperation.’ So I think that if I were their lawyer, that's what I would be advising them.

And I used to be a commercial tech lawyer. So then there's another outcome which I sort of gestured towards is you get a kind of a tiered market for components of any kind, whether it's tooling or models or whatever, where some providers think it's worth their while, in order to capture the market space of those who want to provide high-risk systems, that they will provide that undertaking gratuitously in their standard terms. They think it's worth it if they see a market share for them. Or they might provide certain models and certain components on those terms and others not.

So we get a sort of a tiered market with the compliant stuff and the cowboy stuff, and you pick and choose. And obviously some high-risk AI system providers won't do that due diligence and they won't appropriately select. But I think that will have a positive regulatory effect. Another outcome which we might see is — so the imaginary in this situation is of a sort of a more simple linear development process, and it doesn't fit the complex development processes. But maybe we adapt the development processes for high-risk AI systems.

So in order to be compliant we do just build them bespoke. And there's a lot of good reasons to adopt sort of data-driven, ‘small AI’ solutions for high-risk applications, including privacy, including cyber security, including performance sometimes. And this just adds to that list. It's also easier to comply with regulation, and sort of the management of responsibility in those high-risk situations becomes much easier if you adopt those more linear, more contained development approaches. And then a fourth market solution we might see — and I think that you sort of see signals from Microsoft Azure and some other providers — is that middleware providers, large cloud service providers, are well placed.

They have the bargaining power to actually provide assurances to their customers and provide compliance as a service, which they sort of do with privacy anyway. They're used to that. So what's next? There are two things I want to say. One is, having dunked on the AI Act, I just want to suggest one small change that would make big inroads on this problem. I don't think it is the solution to AI value chain complexity governance. But if we're willing to [inaudible] hold our nose and say an app provider who, whether they like it or not, gets their app modified into a high-risk AI system is then enrolled in the job of assisting the modifier with compliance — if we're willing to do that, why can't we just do that for everyone?

Everyone has a reasonable endeavours obligation. If it so happens that your component — you can't contract out of it — if your component ends up in a high-risk AI system, you have to use reasonable endeavours. Lawyers know what reasonable endeavours means. It's proportionate. You don't have to do everything, but you have to help out. You have to cooperate. I think that if we're willing to essentially impose that on this very narrow sort of tranche of the value chain of sort of modified apps, it's sort of arbitrary not to distribute it more widely.

Maybe we don't distribute it to everyone, but we can make that a longer list. But ultimately, the bigger-picture issue here is that private ordering through contracts, or through this sort of process that we rely on where someone sues a manufacturer in product liability and then they join their suppliers in the action and they sort of duke it out and their insurers duke it out — it is not the right modality. It is not the right conceptual framework for networked interdependency. Contracts are generally tools for passing responsibility.

Of course, they coordinate action to some degree and they provide a structure for relationships. But any lawyer will tell you that the most heavily negotiated provisions in contracts are the limitation of liability and the indemnities, and maybe the IP ownership. And large technology companies are not known for willingly assuming responsibility for costs that they can push onto someone else, and they have the leverage almost in every circumstance to do that. So the idea that we're securing through contracts, through written agreements, compliance as a main way of dealing with this — and that's essentially by default what would happen in Australia absent regulation — is not really an attractive prospect.

And the status quo in our legal framework, in our liberal Anglo-Saxon framework, is very much that no one is their brother's keeper. Responsibility is very atomised and it's very tightly linked to very sort of linear Newtonian narratives of causal responsibility. If you created the risk and it's really clear that you're in a sort of direct relationship with the person who suffers from the realisation of that risk — yes, then you have responsibility. Or if you sign up to that responsibility by contract, yes. But the idea of a sort of a duty to rescue or a duty to collaborate is — you can find sort of little hints of something like that here and there in some doctrines, but it's really sort of anathema to our way of thinking.

So what I'm interested in thinking about is making a sort of a shift out of doctrinal work, legal scholarship, towards a bit more theoretical work, and thinking about more theoretical foundations for an idea of mutual responsibility in an age of networked interdependency. And if you're interested in that, I would love to talk to you about it. Thank you.

Greg Sadler

Thank you so much. We've got one question. So maybe two in the app. And if people have questions, just raise your hand and we can talk a little bit. So the first question was, is there realistic access to justice for an Australian harmed by AI today, considering those points you were making about the web of contracts and maybe technical uncertainty between models or the involvement of overseas companies?

Henry Fraser

Yeah, that's a great question. I think that when we think about the allocation of responsibility by law, it's helpful to split into two concerns. So compensation or recourse — I actually do think that product liability and so on will work out. The problem is not necessarily the Australian consumer. It's the Australian kind of middleman app provider who is going to land with all of this liability for things that they can't control. So if we want to be sort of jingoistic and protective, there are reasons to try and spread responsibility.

And then also if we want to secure better risk management, there's no point in placing all the liability when the risk eventuates on somebody who can't do anything about it. So it serves the objective of compensation, but not of deterrence or prevention.

Greg Sadler

Yeah, yeah. Understood. So another question is, how realistic is contracting as a solution when there's such enormous power imbalances between the offshore AI company, maybe the small Australian business or the individual?

Henry Fraser

Yeah. So I think it's not realistic unless we're willing to change the way in which we think about sort of arm's length business negotiations. We have consumer law where we're willing to accept that certain kinds of contractual practices, where there's a real asymmetry in bargaining power between a business and an individual, and also between some large businesses and smaller businesses under a certain size, we are willing to intervene. The government is willing to intervene in that sort of private negotiation. So only if we're willing to maybe expand our concept of what an unfair contract might be and see if we can remediate that, would contracts potentially be more helpful.

Yeah.

Greg Sadler

I just wanted to ask a question regarding new theories of responsibility. So obviously you would assume that liability is necessary for people to be held responsible, and so otherwise it would be carelessness in the design of the systems. But would you say that, say a service like emergency AI? Yes. If it worked out that it had less mistakes than human operators, would you say they would be acceptable in utilitarian terms to have the impossibility of ascribing responsibility to any companies? And that way, [inaudible] perhaps the person who's been harmed, not being compensated, perhaps having maybe insurance services coming in?

Henry Fraser

Yeah. So I think that's a great question. It opens up a whole bunch of areas. So insurance, different arrangements for liability. What I'm interested in is essentially the allocation of responsibility as distinct from liability. Liability will often in law attend responsibility and failure. But you don't need to have liability to care about how we arrange the allocation of responsibility, whether voluntarily or through enforceable legislation.

Greg Sadler

Thank you so much, Doctor Fraser, for the talk. Why don’t we give Dr. Fraser an applause.

Henry Fraser

Thank you so much.