AI Scribes in Primary Healthcare Settings

I run a civil society organisation called Digital Rights Watch. We've been around for about ten years. We advocate for freedom, fairness and fundamental rights for all those who engage in the online world. We do a lot of policy advocacy work and campaigning. A focus in recent times has been on privacy reform as well as issues around online safety, but increasingly on AI regulation, which is a huge topic. But I suppose what I would say is [inaudible] what I'm going to be talking about today is much more about people's real-world experiences of AI tools rather than frontier models.

Or perhaps in contrast to the last speaker, some of the more cutting-edge issues that are arising within the context of AI. But I actually think this is very squarely a safety problem. So I take a quite expansive view of what constitutes AI safety. And I think meeting people where they are and how they're experiencing these products is critically important to a social understanding of safety that can be reflected in our policy environment. So Digital Rights Watch has taken an interest in AI regulation. We've been campaigning and working on behalf of communities to advocate for government around improved regulations in relation to AI.

I probably don't need to tell this audience, but Australians rank as some of the least optimistic globally around AI and its possibilities, and concerned and interested in more regulation of these products and technologies, [inaudible] and would have greater confidence in them, according to polling, if they understood them to be better regulated. That is also a problem that is becoming more pronounced over time, and particularly since the decision to move away from hard regulation of AI towards a national plan that relies on regulators enforcing existing laws.

But also, I suppose, more general policy pronouncements rather than enforceable obligations, or, for example, transforming the guidelines into enforceable obligations has not seen an acceptance of that as the regulatory norm that should stand. And in fact, polling suggests that Australians are becoming more concerned about the failures to implement hard regulations, and becoming what the pollsters describe as more extreme — as in, their attitudes are becoming sharper about the need for regulation, in a context in which that regulation is not being pursued by government, which I think is an interesting phenomenon.

So telling people to trust these systems, or that they don't need to be regulated in the way that people expect, is not actually working on the polling about how people experience and use these tools. So one of the jobs that we gave ourselves as a civil society organisation was to examine and seek to surface some of the harms that might be arising from community members in how they deal with AI technologies. And we've conducted quite a broad and detailed environmental assessment of where that might be and what issues are of concern to people, what has not been looked at in the way that we would like to see.

And there are, of course, existing regulators, as we all know, who examine some of these topics. But the first issue we thought we would examine, for a variety of reasons, was the use of AI scribes in primary health care settings. So people may be aware, but these are being taken up at an extremely rapid rate. So take-up has doubled, I think, in the last 12 to 18 months, although the data is not entirely clear. So about 40% now of GPs use some kind of ambient scribe.

There's less data around how that is used in non-primary health care settings. And I think there are problems that are probably more acute, particularly for specialist care and particularly for people with specific vulnerabilities, or like children, for example. But our decision was to confine the topic to GPs in primary health care settings, because it seemed like a good way to identify some of the issues that might have application more broadly. So what we found is that AI companies are exposing Australians to healthcare systems which are unregulated, untested and often without consent.

So the broad uptake by clinicians has not seen any correlating significant changes in approach to how people deal with this when they're in these settings. So we conducted a range of research, including looking at privacy policies, including looking at how providers talk about their products. We conducted a bunch of consultations with healthcare users and community organisations that advocate for health consumers. We also talked to general practitioners and their peak bodies about how they use these tools. And overwhelmingly, this is an unregulated space. The TGA has some responsibility for this, is conducting a review, but has elected not to regulate these products.

So it doesn't offer any approvals for by the TGA. For scribes, this is in contrast to other settings in places like the UK where that does occur. For example, the OAIC, the privacy regulator, does not have any specific guidelines or regulatory intervention in relation to scribes. So what we found: we reviewed 60 privacy policies of medical practices that are spruiked as using these on the provider websites. And only a very, very few of them had any kind of indication that they used these. So only four of the 60 that we reviewed referenced AI scribes.

13 of them had no privacy policy, and only three of them referenced audiovisual content collection. So very low levels of compliance with privacy law. And the effect of the TGA seeking not to regulate means practitioners who want to use these tools also have no way of assessing how the product on offer measures up against other products in the market, or indeed how it measures up as opposed to other ways of collecting information from patients. Now, these are real-world problems for patients. The effects that it has had on people accessing primary health care are significant.

So we talked to a lot of different people about their experience of having their appointments transcribed. There's a chilling effect. As you can imagine, there are genuine difficulties when English might not be your first language. There can be bureaucratic nightmares where you're seeking to access something like the disability support pension, and there's an error in your transcription that becomes very difficult to undo. And we've also done a review of different studies around the world of the ways in which [inaudible] AI scribes are performing poorly. And there are a lot of errors.

So people may know about this study already, but AI note-taking is used in social care settings. So in the United Kingdom, when people were being assessed for social care needs, there was a differential treatment of women versus men based on reporting pain, which has an impact then on the thresholds and the type of care you would receive in response. There's all sorts of what I would describe as hallucinations or errors in transcription, including things like identifying cancer but in the wrong breast, identifying someone as having a diagnosis when the note was that they didn't have a diagnosis.

Missing words is a huge problem. Failures to record social history in a context in which that was quite important to the diagnosis. So a range of very complex problems that I think are largely disguised by the lack of regulatory intervention at a federal level, but also within specific regulators with responsibility over these tools. Now, that can be very difficult for patients to identify at a structural level because their experience of it is individualised. So they rely on organisations and researchers to identify these problems. It is also difficult for clinicians who have no way of assessing the trustworthiness of these products and are reliant essentially on industry to do that.

So our report — it hasn't yet been published, but it will be very shortly. But I thought I would give you all a sneak peek. It makes a bunch of recommendations about these problems and surfacing them as real problems that need to be addressed, but then also recommendations for different players within that field to improve the experience of AI and the quality of the technology, understanding it as an optional extra rather than something that has to be integrated into practice as well. So we've made a number of recommendations for four different groups: for regulators, for government, for civil society, and for industry and clinicians.

You can probably imagine what they are. But for regulators, at least some of this should be obvious. From what I've said, we do think the TGA needs to regulate these products and ensure that there's some standard around quality, and to give clinicians a better set of guidance around how to make good choices in procurement. We think the OAIC needs to investigate this field because there's clearly a lack of compliance. This is a problem, [inaudible] particularly in a low privacy law environment, which Australia is, where people have very few rights.

That data is very rich and valuable. And our concern is that improper regulation of it is leading to all sorts of associated problems, including monopolisation, but use of people's personal information in ways they didn't expect. And we also do think there needs to be a change in how regulators are resourced to deal with these products. Which leads into my recommendations to government. The Safety Institute, I think, is a great initiative, but it has a lot of work to do in a lot of different fields. And assuming that regulators are resourced and capable to take on extra challenges that come from novel products in this way, I think is unrealistic.

So I think regulators need to be properly funded and resourced to look at these issues. And we also think that the Attorney-General should prioritise tranche two of privacy reform. As you can probably imagine, we think industry and clinicians need to do more work to instil trust in how these products are being used. We think there needs to be a better understanding of the rights that are owed to patients, including a right to refuse this kind of technology, as well as improving concepts like informed consent. And lastly, we think there's a real role for civil society to play in elevating some of these harms, understanding how they're affecting community members who are left out of some of these contractual relationships between clinicians and providers, and that there's real work to do in researching and understanding these problems, and then taking them to the people who need to act on them.

And so we intend to do more work of this nature, but we would encourage and welcome anyone who wants to work on these issues with us, looking at finding other ways to elevate different kinds of harms that are experienced by people in everyday settings, to ensure that the AI industry is working for those people, and not just for industry and government. Thank you.