What are the top risks from AI?

Alexander Saeri

So my name is Zan Saeri. I'm a director of the AI Risk Initiative at MIT. MIT FutureTech. So this is a joint initiative between the business school and the computer science and AI school at MIT. And really, what I'm wanting to do today is talk about what are (based on some research that we recently released) the top AI risks. So if you take nothing else away from this discussion: just read this slide and take a photo of the QR code. We conducted a Delphi study of more than 270 international AI experts.

And they predicted that over the next five years, the most severe harms are going to come from cyberattacks and weapons (so weaponisation of AI), AI possessing and being used, for its dangerous capabilities (such as persuasion, political strategy, recursive self-improvement, bio uplift, CBRNE), the competitive dynamics between companies and states (that lead to the erosion of sort of safe and responsible governance of systems), power centralisation )where benefits to AI, accrue to those who already have access to the models, as we've seen very vividly recently around mythos), and the provision of and widespread dissemination of false information, misinformation and disinformation.

So I'm going to kind of unpack this in a lot more detail, but I really encourage you to take a look and explore the findings on our website. We have lots of interactives, and ways for you to engage with it. It's not just a PDF on arXiv. I also just want to flag that what I'll be talking about today is just one piece of a bunch of different streams of work that we do. At the AI Risk Initiative, where generally speaking, our mission is to try and help decision makers prioritise, and manage risks from AI.

And so today I'm going to be talking about just, this piece here about sort of the high priority risks, but we actually have a bunch of other pieces of work trying to understand what are mitigations, what are the promising mitigations, where the real world harms already happening, how AI companies and other large companies responding to these risks and, what more exists and what's the coverage of this law, all of these governance approaches. And so we try to kind of combine these together now. I think that after this morning, what everyone needs is one more taxonomy of risk.

But, this one is ours. So I'm biased. Essentially what we did was we tried to understand what were the different risks from AI. And when we started this work back in sort of early 2024, we found many different frameworks, but none of them really had full coverage. There were ones that were talking specifically about the kind of catastrophic risks. There were ones that were talking about, kind of risks within different sectors like financial risks, medical risks and so on. And so what we did was a synthesis of this work.

More than 70 different papers in order to build a database, which again, you can explore freely. And this, taxonomy of different domains of risk. And what this sort of shows us is that there are many different risks. Right. But that makes it difficult then to understand where should we be putting our limited attention, our capital, and even sort of the political capital that we might have. You know, we have risks like discrimination. You have risks like AI fraud and scams. And even, as we've seen recently, the ability for AI to provide very significant uplift, in cyber attacks or cyber security (that was, of course, the sort of, pretext for, disclosed reason as to why, the access to Mythos was shut off last month).

So across this large and broad landscape of risks, we argue that it's really necessary to prioritise. Just because there are many different kinds of risks that doesn't mean that they're all going to have the same impacts. They're not going to harm the same number of people, or have the same impacts on privacy, or other sorts of intangible harms. We need to allocate the scarce resources that we have access to. And also, importantly, we need to understand who it is that is likely to be harmed if these risks materialise,

and who should be expected to act to address them. We might all have intuitions about this or, have spheres of influence or, I guess priors on how we should approach these questions. But we really wanted to try and understand: what does the evidence say? So we try to take three different lenses on prioritisation here. I think kind of borrowing from and adapting from this idea, which is: we're in a period of extreme sort of strategic uncertainty and the evidence that is available to us is often quite poor quality,

and we don't always know what we need to know even 3 or 6 months in advance. So instead of having only a single frame of prioritisation we chose three different frames. So the first was which risk do we think has the most severe expected harms. And so there are many different domains of harm here. But just as an example might say, you know, how likely is this domain of risk to cause a certain amount of damage over the next five years? We also try to understand

who is most vulnerable. So if these risks materialise, who's going to be harmed the most? And we looked both at different actors in the AI ecosystem (for example, developers versus users or the public), as well as different kinds of sectors (so finance, information, agriculture, manufacturing). So and then finally we asked who is most responsible, who should be expected to, to address these risks or work to mitigate these risks in some way. So you can think about like severity is “how bad will it be when it happens?”, vulnerability is “who will be affected most if it happens?, and responsibility is “who should make sure it doesn't happen?”.

As I said, we work in an extremely uncertain landscape. And so, although in the past we've used systematic reviews and other ways of synthesising existing research to try and prioritise the landscape, here we used what's called a Delphi study. So we recruited more than 270 experts across many different domains of risks and also from different industries and sectors. And over three rounds of a Delphi survey we presented to experts. Here's the set of risks: “which of these do you think is most severe?”, “which of these people do you think are most vulnerable?” and so on.

And so importantly, the difference between the Delphi and the typical survey is that experts here are not just giving their one and done impression. So an expert who had expertise in a particular risk (let's say, for example, cyberattacks), they got to give their judgment of how severe that risk would be, they got to give their judgment of which sectors were vulnerable as well as who is responsible, and then importantly, we aggregated all of those results among the other cyber attacks experts and presented it back to them in the second round.

And then we did that again in the third round with the idea being that people can learn from the ratings and the reasoning that has been put forward by experts in previous rounds. And so ideally you would get some amount of consensus or at least stable lack of consensus about what risks are the most severe or who's vulnerable. So I want to talk about these three different lenses. So the first lens is “which are the most severe expected harms?”. And I'm going to gloss over a lot of the kind of rich methodological detail.

I'm very happy to sort of speak to this, individually or in Q&A. But how we measured the severity was that we said “hen it comes to this particular risk (which an expert was in or it was an expert in), how likely is it that that risk is going to have a certain level of harm over the next five years? So how likely is it that it's going to have minor harm? How likely is it going to be to have substantial harm, severe and catastrophic? And these are just some of the domains or the areas of harm that we used.

We also looked at privacy, civil rights and so on, but we kind of aggregated this a lot together. And so coming back to the headline finding: what we can see is that if we just take an expectation of nothing changes (so we're just proceeding as we are) then we expect that dangerous capabilities, competitive dynamics, weapons and cyber attacks, power centralisation and false information to be the risks that have the highest expected severity globally over the next five years. And so this figure is quite a complex figure.

You could explore the interactive website. But this is what that part of the figure shows, right? So this is showing the average severity for each risk under what we call a business as usual or a current trajectory. One thing worth pointing out, though, is that the average obscures these surprising and concerning findings about the tail of the risk. So it's actually the case that if you just look at the likelihood of catastrophic outcomes, which we described as more than a million deaths, more than $100 billion in economic, loss, and civilisational scale intangible impacts on things like civil, civil rights, privacy, democracy and so on.

Actually, most of the risks had more than a 10% chance of catastrophic outcomes over this period. In addition to the business as usual scenario, we also asked what happens if governments and companies implement what we call pragmatic mitigations. So we didn't we intentionally didn't define this because trying to figure out what the mitigations are is a whole other discussion. But if we just imagine that there was some kind of relatively pragmatic and cost effective set of mitigations that could be put in place, how much would we address each of these risks?

This helps us, in some sense, to get a sense of the tractability of the different risks. And so for most risks, pragmatic mitigations make the expected severity go down. But many of those risks that were the higher severity are sort of stubborn. They still remain more than a 10% chance of catastrophic outcomes even under pragmatic mitigations. And so this suggests that we're not really talking about things that are going to be able to be quickly or or technically fixed by like a small number of entities.

These are more structural issues. And you can already see some resonance with some of the previous discussions that we've already had today, including Beba’s talk where obviously we have kind of, technical or model level things like capabilities (and the use or the misuse of these capabilities), but we also have systemic or societal level issues (like power centralisation, inequality, and unemployment, competitive dynamics and so on). So when it comes to severity, we can see that there's many of these risks that are seen as having quite significant severity and a subset, seen as having catastrophic severity or, are quite stubborn to mitigations.

Let's take the second lens here. So the second lens here is who is it that's vulnerable? And I'm going to put a very complicated set of text up on the slide the moment you don't need to read it in great detail. But essentially what we tried to do is we said “Okay, experts, we have a whole bunch of different kinds of ways of thinking about the ecosystem, what if we distilled it down to the absolute bare minimum of what we call actors?” (so different sort of entities in the ecosystem).

And then also there are different kinds of sectors in the economy. And so for each of the actors and the sectors “how vulnerable are they to a particular risk?”. So we might ask a question like “when it comes to false information or when it comes to toxic content, how vulnerable is this particular actor or this particular sector?” to answer that. And so I'm just looking at the sector side of things first. What we can see is that information, finance, and national security were judged by experts to be the most vulnerable sectors across the set of risks.

So this figure you can see here is ordered by severity. So it's in the same order as the figure you saw before. Each row is a different risk and each column is a different sector. And so red is more vulnerability. And the burden or the vulnerability of sectors is not equally distributed. Right? So it's not the case that every sector is vulnerable to the same risks. And it's not the case that for a given risk every sector is equally vulnerable. And in fact, in many of these cases, we were able to establish

over these rounds of the Delphi consensus on these positions. So the dot indicates that the experts were able to reach consensus. So there was consensus, for example, that information was extremely vulnerable to dangerous capabilities, power centralisation and so on. But it's also important to consider that for many of these risks, we weren't able to reach consensus. And this suggests that we have to be, I guess, quite careful about how it is that we respond to and act upon this work because it's the case that, if we don't have consensus, then that suggests that either: perhaps these impacts haven't really occurred yet, perhaps there's meaningful sources of disagreement about how that risk might materialise or emerge within that sector.

I just want to caveat the results with that. So we've talked a bit about severity and then we've talked about vulnerability. Let's talk about this last lens: “who is responsible?”. So here, I'm just presenting a kind of different way of talking about these actors in the ecosystem. Please don't focus too closely on the arrows (you know, “should the arrows be double headed?” or “should they be in this order?”) but here's a really simplified kind of AI ecosystem if we're thinking about finance as a sector. So you have AI infrastructure providers that provide the cloud compute or the training data, that is used to train general purpose models by general purpose developers.

Sometimes these models are further adapted or refined by specialised developers, sometimes specialised developers just don't use a general purpose model at all. You also have companies or entities that deploy AI without developing it themselves. And then you have entities that just use AI. So, for example, if you're in a company that has Copilot it's likely that your company would be considered to be a user of AI in this way. And of course, you also have people who are affected by the decisions or the actions that this whole ecosystem kind of takes.

So, if you’re being denied a loan application on the basis of someone in a company who has used an internal deployed model then you'd be an affected stakeholder. And across a cross-cutting element here are governance actors, which includes government, but also other specific regulators, peak bodies, really any entity that has a governance role. So the reason I'm kind of unpacking these different elements is that when we ask experts when it comes to these different risks, who is most responsible for addressing them and who is least responsible for addressing them, it's a pretty clear pattern.

So here you can see that experts hold general purpose AI developers, as primarily responsible for most risks. But at the top end of the severity, there's a mix of general purpose AI developers and governance actors. And importantly here, just because I've put them here as most responsible, doesn't mean that experts said that the other actors were not responsible at all. I'm just simplifying it for the illustration. And again, as a pattern for almost all of these risks, it's the affected stakeholders that are seen as least responsible.

Now, this might make sense because we say it's the case that we shouldn't expect the people who are impacted by these decisions to have responsibility for addressing them. And it's reasonable, perhaps in some sense, that the people who, have the most capacity to address these issues, are more responsible for addressing them. But the problem is that that only really works if we can expect that the people who are less responsible are also less vulnerable, and the people who are more responsible are more vulnerable, because that would be where they'd be, as Beba was talking about, an economic alignment in incentives.

So it might be the case that if you are highly vulnerable, to a particular risk, then it's okay for you to be responsible because your behaviour is aligned in this way. But if you have a lot of responsibility, as judged by experts, but you're not actually vulnerable to this risk then potentially you're going to under invest in this because you don't have to bear the cost of making mistakes or incorrectly managing risk. And then, conversely, if you are, not very responsible, you don't actually have much power in the system to change things and you're very vulnerable,

then you're kind of dependent on others. So what does the data show us? Well, this is aggregated across all risks, just for the purposes of illustration here. But what you can see is that you have, the people who are most responsible are not vulnerable and the entities who are most vulnerable are not responsible. And so essentially this means that the developers who are, or to some extent the governance actors who are in a position to, actually affect these things are more protected from the impacts of the risks than the people who are going to experience them.

And so really, this suggests that the public might be in a position where they're experiencing the harm. Okay, let me just wrap up. So what does this all mean? This data suggests that not all risks are equally important. And also, importantly, many of these risks are associated with potentially catastrophic outcomes. And some of them are resistant to, you know, even sort of typical mitigations. Some of the higher severity risks might require structural change just by the nature of the way in which they manifest. Right? It's not the case that a particular company that is exposed to, let's say, for example, competitive dynamics can just unilaterally decide to slow down their adoption of AI because their lunch is going to get eaten by a competitor.

Some sectors, as we've seen, are especially vulnerable and ultimately, the people who are most responsible, so who are most vulnerable are not in a position to be able to address it for themselves. So I just want to zoom out then and create some time for Q&A just by kind of pointing out some of the things that we're doing. Next, around these work to try to take this and actually do something with it. So we're looking to try and build a database of mitigations. We're trying to increase

the reach of our incident tracker, and we're going to be focusing more and more on the risks that actually have been identified by experts, as priorities. We're also looking to update this prioritisation, because we conducted this in late 2025 and everyone knows how quickly things are moving. And so we're hoping to update that, this year as well. And just as a sneak preview of some of the work that we're finalising at the moment: over 200 companies, 200 of the largest companies, we tried to assess what percentage of the companies were actually responding to different risks.

And so the coloring here, again, is their vulnerability. But as you can see, for example, Information Systems is responding to, you know, between like 20 and 30% of the companies in our sample are responding to risks. But this is again sort of in preview. So I'll just finalise by saying, please come and talk with us. Over the course of the conference, myself and my colleagues, Michael and Quinn here, very keen to talk with anyone who might want to use our work, or, be a contributor to future rounds of the work.

And again, please feel free to explore the findings in much more detail. I'll leave it there. Thank you. Thanks.