Who Determines What Safe Is? Gender, Diversity and Power in the Future of AI

My name is Johanna Weaver. I'm the executive director of the Tech Policy Design Institute, and this panel is going to be 25 minutes of power, looking at the questions of diversity in AI safety and particularly looking at how do we define what harm is, whose harms matter. And I think that's a really important question. And then who gets to decide which harms matter as well. We have some pretty amazing panellists with us today. We have Doctor Gemma Killen, and she's the executive director of Working with Women Alliance.

She focuses on gender equality and has done recent research looking at how AI can affect women's economic participation, health outcomes, and the equitable access to the benefits of AI, which I think is something that is increasingly becoming topical in all conversations that I'm having. And then we have Professor Rashina. Professor Hoda is from Monash University, and her work focuses on human-AI collaboration, AI bias, sociotechnical factors that shape safe and trustworthy AI. A pretty well-qualified panel to be talking about this. Now, I've really challenged them because we've got a lot of questions and I've said, you've got one minute for each answer, so bear with us as we provide high-speed answers, and then encourage you to engage with them in the break or afterwards to delve a little deeper into all of these issues.

Our first question is, I think, a really important one, which was covered in a lot of the plenary sessions, and that is what counts as harm. So when you hear AI safety, Gemma, what's the first risk or harm that comes to your mind? And what do you think is one that is currently under-recognised, that maybe wasn't discussed in the plenary sessions that we've just come from?

Gemma Killen

I thought about this question a lot, and I think it really depends who you're talking to, what comes to mind first. In my circles, where we talk about safety and gender equality, everyone is really across the risks in gender-based violence and the ways that AI is escalating gender-based violence. And I sometimes forget that the broader community isn't as across those harms and those risks. Something that we're keen to progress in this space is making sure that women aren't left behind in AI transitions. We're now talking about job augmentation instead of replacement, but I think that we don't talk enough about what that means in terms of who the harms are shifted onto and who carries the risks around augmentation.

So that's something we're really interested in talking about more.

Johanna Weaver

And what about for

Rashina Hoda

you, Professor Hoda, you can just call me.

Johanna Weaver

I like the professor. You've earned that title.

Rashina Hoda

This is going to be a bit of a weird one, but I hope it makes sense. The first word that comes to mind for me when I think about AI safety is a toaster. How many people used a toaster this week? Okay, they're in our homes. A toaster, slices of bread in the morning routine. Do you know the amount of regulations a humble toaster has to go through before it enters our household and our lives? There is electricity regulation. There is thermal safety regulation. There is regulation around not being able to tip it over by a pet or a child to cause harm, and around fire prevention.

And once all of that is passed, then there is a certification. And when you bring that into your home and into your lives, you know that this is safe. How would you feel now about a toaster that abruptly and randomly was known to shock women four times more than men, or to harm young adults using the toaster abruptly 15 times more than the average human being? Would you be even thinking about such a ridiculous product or concept alone, bringing it into our lives? And yet we're living in the same world where this toaster has to pass all of these tests, where AI is in our lives, in our houses, in our pockets the whole time, displaying this very kind of behaviour which is harming

vulnerable populations disproportionately than others.

Johanna Weaver

I think of the toaster — I love that, I think it's very powerful. And it really emphasises why this is important for us to be thinking about it. So the next question I want to ask about is, are we asking the wrong questions in AI safety? A lot of the conversations that we talk about around AI safety are what are the harms or what are the biases? And we've been thinking about whether or not we should be asking who is answering those questions. Is the AI safety community representative of the population that AI safety needs to serve?

And if it isn't, what are the consequences of that? So, Professor Hoda — and I'm going to stick with the full title — your research shows that AI systems can reproduce narrow assumptions and stereotypes, while Doctor Killen, your work highlights how AI can reinforce inequality in areas like employment and healthcare. So does diversity simply mean fairness, or does it fundamentally change the types of risks that we engage with?

Rashina Hoda

Sure. Thanks. So something that I kind of put out there, also in my weirdness, is this idea that AI is the average of the human experience, and there are no average humans, right? So when we talk about AI, or the people who are designing and building and governing, and the leadership behind AI right now — people heard of the term MANGO? Okay. So this is an acronym doing the rounds around Meta, Anthropic, right, Google, OpenAI space, if I'm missing any. You get the gist. So this is an Nvidia.

So the MANGO kind of represents that top leadership around AI and what that looks like. And it has a very strong cascading effect into the design and development teams that we see in software around the world. So that is a problem in terms of people who are in the places of power and making decisions and designing these systems are not diverse enough to develop something that is claimed to be universal. Right? So if you want to develop something that is universal, it has to be universally safe, as in for all people.

And that starts with the diversity of the team designing these products and services.

Gemma Killen

Totally agree. And I think it's worth pointing out that only 1 in 3 AI developers is a woman, and women are not homogenous.

Johanna Weaver

I'm surprised it's that high.

Gemma Killen

And maybe that's a global figure rather than an Australian figure. We know that the Australian workforce is also more gendered than the global workforce, so we might have particular problems here. And there are big gaps in what people see as the harms, as we've talked about. And while we're stepping back from AI for a second, the Australian government is doing a systems audit at the moment and looking at safety by design across mostly our social services systems. But that same process isn't happening in the development of AI, so we don't have the expertise to tell us what might happen or how people might use these products in ways that are deliberately harmful or accidentally harmful to a large amount of people in our communities, and we don't have mechanisms for accountability or complaint when they do cause harms.

Johanna Weaver

We're going to come to the question of how do we address that. How do we bring more diversity into those conversations? But before we go there, I just want to have one more question, focusing on whose harms matter. We've spoken a bit about the nature of harms, but do you think there are harms from particular members of the community whose harms aren't being given enough attention at the moment? Doctor Killen, why don't we start with you?

Gemma Killen

I think we know that people who are experiencing marginalisation are experiencing lots of harms in relation to AI. One of the things that's come up in my work a lot is the way that AI is used in the Family Court, and in response to sexual harassment and sexual assault claims. We know that the Fair Work Commission is clogged with vexatious claims and the Family Court is clogged with vexatious claims. So you see perpetrators of domestic violence who are using AI to make a huge volume of claims and draw out processes within the court.

And that's not something that we talk about a lot — how something that we imagine might be neutral, like an AI chatbot, is used or weaponised in these ways to clog up our justice and policing systems.

Rashina Hoda

In terms of whose harm is missing, if I could just maybe request you to literally look around for a second, right? Just look around you. Yeah. Now, who do you think is missing in this room? Do we see kids? Do we see young adults? Do we see older adults? And then there are things that we don't necessarily see. So there is visible and invisible disability. There are people potentially from low socioeconomic backgrounds. There are people with digital and AI access issues. They're not in this room, and the list goes on.

And those are the people whose harm is, I guess, really important for us to focus on, because that's also the harm that is missing from the data that the AI is trained on. So it is what is around us. It's in the rooms where decisions are being made. It is in the data which powers the AI models. It is in these conversations we are having about AI safety, where these people are missing and whose harm really matters.

Johanna Weaver

And so do you think that's because the definition of AI safety is too narrow?

Rashina Hoda

point. And I think the talks earlier really set a very good shared understanding of where we are standing as a global community right now. And I looked at the malicious use and the technical failures and the systemic risk — the three things that I talked about. And I kind of wondered whether it's not just preventing malicious use, but it's also, as you just said, the unintentional harm, right? The accidental harm. And it's also not just the technical system failures, because AI — and responsible AI in particular, safety in AI — is a sociotechnical issue.

So it has to be looked at from a sociotechnical lens. And it's not just technical failure, it's sociotechnical failure. And the third thing is it talks about systemic risks. But again, I think it's worth highlighting that this risk is for all populations and particularly the vulnerable populations. I think that those things actually need to be spelled out.

Gemma Killen

I think the only thing I would add is that there's a certain level of social acceptance of some bias and harm that is carried over into AI. The conversations that we have sometimes with those in power and making decisions say, well, we can't do anything about [inaudible] the use of AI in employment practices having bias because people have bias in employment practices. So there's not really a difference. And we don't take on what it means to escalate that bias or embed it within hiring systems.

Johanna Weaver

Yeah. And I think it's also challenging ourselves. This is an opportunity to make the world a better place. So to not accept that that is the standard, but also to recognise that we can move on. One of the things I'll just mention is that at the Tech Policy Design Institute, we've recently done a national assessment of Australia's AI capabilities. And one of the things that really stood out — this is 103 different capabilities. But we did look at these particular capabilities that are spreading the benefits, inclusive and discerning AI adoption, etc., and we don't have enough focus on making sure that we are being more inclusive in the way that we are spreading those benefits.

There's an intention from the government to do that. I think it sort of behooves everyone in this room as to how do we actually do that? It's a really, really difficult question, which is where I want to go to next with you. If we — and I think there are a few people who would disagree with this — that [inaudible] when we're looking at questions of AI safety, it's not just technical, it's sociotechnical. Who deserves to have a seat at the table? But more importantly, because I think the answer is everyone representative, how do we get them at the table?

And importantly, how do you make sure that you don't just have people at the table, but they're actually included in the conversation, because it's not enough just to have them at the table. So Doctor Killen, let's go to you first.

Gemma Killen

I think I always come back to this little fact. Men are more likely to have their training paid for by the government or their employer, and women are more likely to have to pay for their training themselves. And when we talk about who's at the table, we're talking about facilitating participation. And paying for participation is one way that happens, and women are less likely to see that. So we have to make sure that we pay women for their expertise, and we design consultation mechanisms and participatory pathways that acknowledge caring responsibilities.

Men also are more likely to access training that happens during work hours, whereas women have to access training outside of work hours, which conflicts with caring responsibilities as well. There are these built-in structures to how we invite people to participate in decision-making that exclude women, and we have to address that as we bring people to the table.

Johanna Weaver

And a quick shout out to the forum organisers, who have used funding from the Department of Industry, Science and Resources to support a women in AI mentoring program. So we have a number of women who are here sponsored to participate in this program, who we've matched with a number of amazing mentors, many of whom I see in the room, but also our two wonderful speakers. So over to you. Thank you.

Rashina Hoda

I'm just going to build on what you were saying around [inaudible] women paying for themselves. I think there's another aspect. There's a very important economic aspect, but there's also the cultural aspect. And one of the things that we're seeing right now is what's being termed as the AI competency penalty. And data shows right now, women are not engaging with AI as much as men, and there are reasons behind it that I'm not going to go into, because of time pressure. But the other flip side of this is that when women do use AI to improve themselves as professionals in their workplace, this is seen as proof of their incompetency.

So I knew what you're going to do it without AI, right? Whereas this with the same level is not meted out to men, who are then seen as, oh, that's a smart person. He's on with it. He's on with what the times is moving with the new technology. So I think that's a personal kind of reflection for us as well, to really take those kind of blinders off and look at it more progressively around not just women, but any kind of vulnerable or marginalised identity. Using AI is not proof of their incompetence.

It's proof of the fact that they are moving with the times and putting effort into their own development. So I just wanted to point out the cultural aspect of this as well.

Johanna Weaver

So I'm going to put you on the spot now and say, imagine you're at the cabinet table, because it is coming up to MYEFO, which, for anyone who's in Canberra, it's the middle of financial year, it's a really important time in Canberra. It's when the second round of money gets handed out. So everyone's focused on MYEFO at the moment. Cabinet needs to make a decision. There's one project that they can fund to make a difference in terms of inclusivity in AI safety. What program are you waiting for to be implemented?

Let's go with you first, Professor. Cool.

Rashina Hoda

Mine's actually going to go back to the toaster. I'm going to stick with the idea that no AI product or service should be allowed to be rolled out to end users without certification and proper regulation fit for the purpose, not just general-purpose anything. And especially pointing out [inaudible] this incredible problem of AI psychosis and in therapy-based apps. Right? So before we start to get into our deepest, darkest secrets with this anonymous entity behind the screen, that we know that it is regulated and that we are in safe hands.

So, yeah, the regulation aspects

Gemma Killen

Totally agree with the regulation aspects. I think we need investment in upskilling women in AI, particularly in vulnerable or marginalised communities, so that they can be at the table. And I think that requires significant investment from government. And then I think an oversight mechanism that includes women and other marginalised communities, so that we have formal ways to give feedback and report problems as they arise, rather than relying on independent individual complaints mechanisms that often fail.

Johanna Weaver

Yeah, I think that's really important. And I think it's not very well known that the National AI Centre has a page on their website which is expressing how existing law applies to artificial intelligence. We often have conversations about how we don't have this AI act. That web page, I think everyone needs to see, [inaudible] and it's really powerful. And then it also has a page talking about what do you do if there are harms? I'm not saying that's enough. We need to do more. But I think that's a really useful resource for everyone in this room to be accessing.

I'm going to wrap up with a very [inaudible] I'm going to say the start of a sentence, and I would like you to finish the sentence. And then we're going to go for maybe 1 or 2 questions from the room. So five years from now, if we have genuinely succeeded in making AI safety more inclusive, what will be the biggest difference that people actually notice?

Gemma Killen

I think we would have a safety-by-design approach to AI, where we don't wait for harms to arise, and we don't rely on people who might be extremely marginalised to report them. But we can embed processes for addressing harms and for preventing harms from the outset. And I think that requires a testing environment that can identify bias and harms before they occur. And I think that happens in relation to national security. We have those conversations, but we need to make sure we have them about marginalised communities.

Johanna Weaver

Absolutely.

Rashina Hoda

So for me, a success looks like AI being designed, developed and governed by more inclusive teams to begin with, the products and services being continuously regulated because these are changing systems, and that all users of AI of all ages know how to use AI that works for them and not the other way around. So that's part of the training and education aspects as well.

Johanna Weaver

Great. Thank you. Greg, are you going to do questions, or do we take them from the floor?

Greg Sadler

So everyone who's logged into the app, you'll be able to type in questions. If you're in the session, then there's questions you are interested in — you can upvote them. If you want to work on that now, please do. But maybe if someone wants to volunteer.

Johanna Weaver

I'll run the mic. I'm just going to go here first.

Audience question

You have finished up with utopian. What do you see as utopian? What do you see as dystopian reality? If everything fails.

Johanna Weaver

I like it.

Rashina Hoda

Easy question. I think utopian, this is kind of a philosophical aspect that I think about a lot. I think with AI, we often talk about what you were saying before about, [inaudible] oh, bias already exists in human society. What are we meant to do? I think AI presents a profound opportunity to live in a world we want to live in, not to live in a world we currently live in. And I think I'm trying to address both your points by saying, look, this opportunity is if we are aware of this opportunity in every decision we make about AI and what we do with AI, in fact, AI can help be part of that solution.

So if we've got missing data, AI can be part of that solution to simulate data that gets in there and so on. So with that, we can steer away from dystopia, which I'm sure Sci-Fi movies are all full of. Yeah, I know you don't need me to expand on that, but The utopian future is where we have used AI to build a world we actually want to live in.

Johanna Weaver

I'm going to take another question for you, if that's okay. I'm going to take one here, and then I'll go to the back of the room, and you can have one question each, and then we'll wrap up.

Audience question

Thank you for the presentation so far. My name is Francine Cluff. I work in policy and crime policy, domestic violence, perpetrator interventions. But I'm here as an interested party. I'm just curious about, for example, you mentioned the National AI Centre. Is that government, is that private? When you talk about, not checks and balances, compliance, you use the toaster example. Is that going to be government? Is that going to be industry led? And so I think my question is, who do you see as best placed to regulate that?

And if it's government, which has more of a centralised and funded and capabilities, they're very slow. I work for government. I know how slowly the wheels turn. Who is best placed to regulate, etc.

Johanna Weaver

And I'm conscious of time and the panel coming after us. So we're going to end on that question. Apologies for the people at the back of the room. Gemma, do you want to take that one?

Gemma Killen

I think in relation to the dystopian question, I was thinking dystopia is where we put profit ahead of real impacts on people. So I would be really cautious about private regulation bodies, and taking your point about the slowness of government. But I think if we're going to have that accountability mechanism, it has to sit with government, but it has to be well resourced to address that slowness.

Rashina Hoda

Another one, I guess, is the dystopian future kind of thing that we don't want, which some people do suggest is to have AI in control, because isn't it all objective and centralised and global?

Johanna Weaver

No, I love it. Just to use the chair's prerogative briefly, I think we need to look at a different type of regulation. There's a proposal on the table at this term of Parliament around the digital duty of care — harm neutral, technology neutral — that can move with the speed of technology. We need to think more creatively about that, and then we need to fund the regulators to be able to enforce those rules. And folks, we've touched on just one aspect of diversity here, primarily focusing on women.

Of course, diversity is much broader than that, but I think we've had a really excellent surface level, but really highlighting some depth that we all need to be thinking about. I want to thank everyone who's chosen to be in the room for this session. It's really heartening to see how many people there are here and interested in this conversation. This is really important and it makes me optimistic for the future of AI safety in Australia. So let's have a round of applause for our amazing panellists.

Thank you.