Carl Bufe

AI Assurance in Clinical Contexts

The Report treats evidence-based assurance — safety cases, performance logs, monitoring — as a set of risk management practices that are only beginning to take hold in AI, and that remain largely voluntary. I work in a domain where they aren't optional. In pharmacovigilance and clinical systems, AI assurance is mandatory, personally named, and inspected. This talk shares one thing that field has had to settle that AI safety is still circling: the line where an AI system stops being an assistive aid and becomes a regulated component. The moment AI enters a decision that affects patient safety, three things attach at once — a defined intended use, validation evidence, and monitoring for when the model changes underneath you. A named person carries the accountability. Cross that line without them, and it's a finding against a person, not a system. I'll make it concrete with a real failure mode, show why agents make the line harder to hold, and concede what doesn't transfer — importing medical-device validation wholesale is a scope error. What transfers is the principle: a defined trigger, named accountability, monitoring-on-change.