AI Assurance in Clinical Contexts
8 July 2026 · 11:25 am–11:30 am · Cullen
The Report treats evidence-based assurance — safety cases, performance logs, monitoring — as a set of risk management practices that are only beginning to take hold in AI, and that remain largely voluntary. I work in a domain where they aren't optional. In pharmacovigilance and clinical systems, AI assurance is mandatory, personally named, and inspected. This talk shares one thing that field has had to settle that AI safety is still circling: the line where an AI system stops being an assistive aid and becomes a regulated component. The moment AI enters a decision that affects patient safety, three things attach at once — a defined intended use, validation evidence, and monitoring for when the model changes underneath you. A named person carries the accountability. Cross that line without them, and it's a finding against a person, not a system. I'll make it concrete with a real failure mode, show why agents make the line harder to hold, and concede what doesn't transfer — importing medical-device validation wholesale is a scope error. What transfers is the principle: a defined trigger, named accountability, monitoring-on-change.
Recording
Audience Q&A
Ask a question or upvote others.
Loading questions…
Transcript
Hello everybody. My name is Carl. I'm actually a pharmacovigilance expert. I also work in regulatory affairs in clinical trials and medical devices and so forth. I've got a few degrees. A generalist, people would tell me, not a specialist. The AI Safety Report describes evidence-based assurance, safety cases, performance logs, monitoring as voluntary. Unfortunately, in my field that's not an option. We can't play around with that. We need to have set rules. We are accountable for safety of information that is publicly available. Clinical trials running. Adverse events flow from site every day.
Free-text narratives that are coded to standard terminology. These are watched for signals. Patterns are recognised. The patterns paint a picture that goes off to the regulator, instigates changes to products, medicines, and so forth. Unfortunately, AI stepped in. Starting to streamline the processes makes things a bit easier. Here's the line. My field is settled while AI drifts. A human owns the record. It's an assistive aid. It's not something that just takes over. It's very important that we have got three valuable assets in our processes. We define the intended use.
We base everything on validation. And it's extremely difficult to validate AI models. And it's actually impossible. We monitor change. So if the models change, we actually act to initiate change control. And at the end of the day, it is not the models or AI that's responsible, it's a named person. Now this is where things get a bit sticky. The trap is that dangerous failures are silent. The model is validated. Go live. Performs well mid-study. The world moves. The vendor ships an update. Nobody knows about it.
The model still reacts like it's supposed to be reacting, but nobody sees what is going wrong. The answer is wrong. This is a signal that should have been surfaced. And if we do not take the appropriate controls in our line of work, these have detrimental effects. So no longer is it just acceptance. We have to build new safeguards. We have to regulate it in a different way. It's a unique way. These are not written in black and white. Unfortunately, regulators themselves are not keeping up with it.
And that's where I find myself in that field of working with a lot of agencies and patients and safeguards and regulators to actually come up with solutions. Now, agents make things much more difficult. An agent doesn't just sit on the side. It chains coding to triage to escalation. It crosses multiple tasks. My field is currently in my field. Just writes a blanket rule saying this is not allowed. Continuous models are no longer allowed. In our practice, Annex 22 is a regulation that the EMA and European regulators are actually debating at the moment.
It applies to good manufacturing practice, but in our field of practice, [inaudible] the rule is blunt. We cannot have continuous shifting models in our line of work. Now, of course, there's a lot of organisations that want to challenge that debate itself, and that's where I find myself, to actually help them navigate those regulatory expectations. But unfortunately, today, as I'm speaking to you, we can't have a shifting field in my practice area. So a lot of the time when I actually speak to my clients, they're not very happy with the output I provide, but I try to navigate them to a more amicable outcome.
And that is the end of my talk. At the end of the day, we don't assign responsibilities to a model. We assign it to a person. They sign off. They are still accountable for the outcome. Thank you very much.