Embedding AI Impact Assessment into Undergraduate Education

Thank you everyone. And given I'm also talking about intersectionality and things after lunch, I also want to add my acknowledgment to the local custodians. I hail from Yuggera Turrbal lands in Brisbane and I'm in Turrbal country. Yep, and our campuses are across the Kabi Kabi footprint. I'm Erica and that's James. The formal title of the talk is about embedding AI impact. But really what we want to say is: how do we raise responsible future technologists? Because the one thing that is massively missing from all these documents we talked about today is not what AI is doing to education, but what are we doing in education about AI?

How are we teaching these people? Last year I was super lucky to have James with me, teaching into our first-year computer science course. So this is a really short bit on how that actually went for us and what we learned through the process, so that hopefully others can do it too. As I said, it doesn't align nicely with the report. It sort of links into the risk management practices section, but it's worth calling out that ISO standards are not mentioned anywhere in the report, even though there are some very established quality ones that, in my opinion,

probably should be. We know it's racing. AI is doing a lot. We know there are heaps and heaps of risks and harms, and we really need to teach how to do that management and how to build that organisational capability. So we, with the help of James, [inaudible] had a bit of an experiment last year, and we dropped the ISO 42005, which is the AI impact assessment management standard.

James Gauci

Is that the correct title, James? Close enough.

Erica Mealy

Something like that. Anyway, we worked through examples. We had James working them through, showing them how he would do one of these assessments as well. And we gave them a framework that they could actually use — and, as you'll see later, it has actually positively impacted how they deal with AI in their everyday lives and in their studies. So, James.

James Gauci

Hands up who's heard of 42005 before? Some people. Can you tell me what it is?

Audience question

It aligns with ISO 42001, which is the AI management standard for organisations adopting and developing AI. And 42005 directly deals with impact assessment.

James Gauci

That's right, well done. Huge round of applause for our guest speaker today. The AI impact assessment process is a sister standard to the 42001 standard, and 42001 requires impact assessment as part of its system. What it doesn't tell you is how to do an impact assessment. Why is that, do you think?

Audience question

Approachable to everyone?

James Gauci

It's meant to be approachable for everybody to use in their own way. Somebody said it's too hard. Yeah, I get told all the time that governance and ethics and the way that we make decisions and things is just way too hard, isn't it? Oh, isn't it hard to align stakeholders? Yes it is. And in technology, we've been doing this for a very long time. It's called things like human centred design, product thinking, user experience and all. All 42005 is a system for doing those things and getting the feedback that you need from the various stakeholder groups that are impacted by your system, so that you can incorporate those insights into the design of your system in the first place.

Does that make sense so far? Yep. And I'm not going to teach you all how to suck eggs, because impact assessments are critical to the design process, especially in private sector circumstances. We're seeing live in the economy right now, all the people who traditionally conduct these kinds of studies — user research practitioners, product people, project managers who usually champion design, designers generally — they're being continually cut out of the workforce at the moment. And usually design and marketing and communications are the first thing to go in technology when times are tough.

Anybody disagree with that? It's okay if you want to disagree. I really want to know.

Erica Mealy

He really wants a fight.

James Gauci

Ultimately, what we want to do with the 42005 framework is assess the potential impacts on individuals. So each of us as individuals — we've looked each other in the eyes before. Groups — like the group of people in this room, people who are drawn together by a set of shared attributes or a shared place or a shared ideal or what have you — and society. And something that isn't mentioned in this slide is the environment as well, which is also a requirement of 42001. And this gives us the levers that we can use to make sure that AI systems vastly and unquestionably are beneficial to the various impacted stakeholder groups, and that the appropriate controls and mitigations are in place for the risks and the harms.

And that's the kind of world that I want to live in. That's the kind of AI that I want to develop. Trust and transparency, harms and benefits, accountability. Let's talk about accountability just for a second, because we got told that there is a shortage of trust in Australia. Why is there a shortage of trust, do you think? Hands up. Anybody want to have a go?

Audience question

Global trend in reducing trust in institutions.

James Gauci

Yep, there's a global trend decreasing trust in institutions. What else? What about particularly in Australia, where we rank very, very low on trust in AI systems?

Audience question

People don't understand what we're talking about.

James Gauci

Yeah. People don't understand. People don't know what AI is. Yep.

Audience question

I think there's a big focus on harms, including AI cheating, AI hallucinations, that kind of thing.

James Gauci

Yep, yep. I was going to say it's seen as a transformative change, but to what? No one's really defined it — AI is this amorphous concept to most people, isn't it? What has been notably absent in all of the talks that we've had today? There's one particular key stakeholder group that we don't talk about very often in the public conversation about AI.

Audience question

Older people. Yep.

James Gauci

Who else? Kids. Who else?

Erica Mealy

So there's more than one, is what you're saying?

James Gauci

Yes. That's the point. But also there are large multinational organisations with huge amounts of capital making all of the design decisions, right? And if you've worked in technology like I have for 15 years, you know that the technology systems that you build reflect the people and the systems that built them, invariably within a bound of error. Of course, you've got to account for probability and risk in there. But that's also why we want to teach these first years, before they get out into these big multinationals, before they get out the door, that they have to stop and think.

Tech has a terrible habit of not doing impact assessments properly. We've had things like cars that were hackable from the other side of the United States, because no one actually stopped to think about putting a cellular chip on a radio. If we do an impact assessment properly, then we can actually try to mitigate some of those harms. And that's where we came in with these standards. It gives us a process. It gives us the actual documentation that we need to be auditable and to continually revisit.

Erica Mealy

So, how we used it. We worked them through designing a process. We provided them with that process, but then we went through with them and documented those assessments. And it was really interesting to watch them go searching and see how little information is on those safety cards, how little information is available about what data it was trained on, and to really get them to stop and have that half moment of, oh goodness, what am I feeding my data into? But the main areas — and I'll hand back to James — that we got them to have a look at with these ones specifically.

James Gauci

Yeah, it's a huge standard, and I agree it's hard. But if we have no shortage of capital in this country, we need to put resources behind these kinds of systems and practices in order to ensure that AI systems align to the values of the individuals and the groups and the societies that they serve. System, information, data, algorithm, model, deployment, stakeholders — those are covered by 42001. So those are inputs that come out of 42001 and go into 42005. And then there are impacts. They can be positive and negative, and we invariably focus on the negative in AI risk and AI safety.

So it's important that we understand some product thinking. What are the opportunities here? How can we better serve underserved communities? And action plan: what are we going to do within the AI system development lifecycle to actually improve the system continuously?

Erica Mealy

We've only got two minutes, so these are probably things that you're all very familiar with: the idea that the quality of the data dictates the quality of what you get out of it, that we really need to plan for misuse and misadventure with these models. And these are striking moments. For first years, they've never thought of any of these things. So what we keep [inaudible] we had a very complex, nuanced case study. James provided one around industry and mining. We also had other ones around AI creativity.

The plan for this coming semester — we're actually looking at adding AI and automation into gene therapy. And if we can actually bring the cost down from $150,000 a person... So one of those times where maybe doing the person out of the job might be the moral thing to do. So we're giving them these really messy problems. We're working it through with them, with the deep involvement of folks like James. But they and us struggled with, what are these risks? Where does the severity level sit?

And actually looking at the harms and seeing some benefits in there. So I think those two particular sections are the ones that we're going to go into deeper. But these are quotes from two of my students. I went and harassed them earlier in the week. The kids are all right, guys like that and like put yourself in the room.

James Gauci

These are computer science students in their first year. This is the first time, often, that they've gotten the chance to talk about the societal impacts of the systems that they are going to be responsible for in their careers. Erica.

Erica Mealy

Yeah, so you can tell them that I told them what I was presenting it for, because they decided to use words like cognisant. Pretty sure he used Claude to help him answer that. Anyway. So I guess that's our why. That's a bit about the standard and why we've done it and what we would do again. So thank you so much for your time. It's very short. We've got 25 seconds for a question. Yes.

Audience question

What's the weakness of 42005?

James Gauci

The weakness of 42005 is it's too heavy. And if you try to implement all of it, it's probably not going to get past the financial review and cost.

Erica Mealy

We had to buy it for the library, so it's not available for everybody. Yeah. Cool. Thanks, guys. We didn't have time for that.