Will the real FraudGPT please stand up? Dark AI and Cybercrime Markets

Thanks for having me here. It's great to be a part of an event like this with so many different people from the public, the private sector, from academia as well. I'm Doctor Andrew Childs. I'm a lecturer in criminology up at Griffith University in Queensland. So thanks to the organisers for ending this tomorrow so I can do my mandatory State of Origin watching as a Queenslander. We have a technology and crime research cluster up at the Griffith Criminology Institute that I lead as well. So what I'll talk about today, I'll kind of wrap up a little bit more about my research and what I do with some of the intro slides.

But I don't come into this event today as an AI crime expert necessarily, or an AI safety person specifically. But I'll sort of talk about how my research is bridging into that space with some findings from a recent project. So just to anchor this in the International AI Safety Report. As the intro was really helpful at scene setting, what I'll talk about really is around how criminals are using AI — what are we seeing in the space of dark web communities, cybercrime communities, of that malicious use of AI?

I come into this area as an expert in online illicit markets. I'm a massive nerd for online illicit markets. I'll run through the reasons in a second. If we think about our understandings of what a normal market is, we've got a buyer and seller and they're exchanging a product in some way. Same sort of deal. But when we come to online illicit markets, we're talking about either the products that they're exchanging are illegal, or the way that they're exchanging that product is illegal or illicit, if we want to tap into some of those grey areas.

So I am a nerd for online illicit markets for two reasons. One is, if we track the trajectory of cybercrime operations over the last two decades, really what we do see is that online illicit markets have the capacity to increase the opportunities for people to offend. They do that by lowering the technical resources or skills needed to commit crimes. Ransomware and other sort of classic cybercrime cyber security attacks are a really good example of that, right? We've seen the general shift from people needing to be really skilled hackers themselves to just being able to buy scripts and resources to be able to perpetrate their offending.

But the other reason, and the thing that keeps me in a job, is that they are always changing. So much. So the organisation and the operation of online illicit markets is not this one-size-fits-all type of model. They differ so much depending on where exactly we look. So are we looking at a dark web marketplace where the buyer and the seller are using cryptocurrencies and they're exchanging things that way? Are we looking at encrypted messaging apps? Are we looking at more diffuse communities that might be spread in Discord servers or on social media platforms, for example?

And they also change so much depending on what we're actually looking at as the commodity or the object that's being traded in these markets as well. So the way that illicit drug supply happens through digital environments is so different to how the exchange of fraud products or fake identity services work between different digital environments as well. So that's how I kind of come into this space. I initially did not want to be another young, nerdy white guy keen on AI until it's started sort of creeping into this topic that I explore.

The main thing that I started coming across a little bit was this growing concern from national and international policy and policing agencies. So the Australian Cyber Security Centre, the UK Home Office, the FBI, the Europol — around these, what they were calling in a lot of these occasions, dark AI tools, this kind of shadow market for AI tools that cybercrime offenders are using. That is not just the [inaudible] criminal exploitation of ChatGPT and some of these commercially available models, but these are shadowy figures. So we know then that offenders are using AI in lots of different ways.

But the thing that really brings me into this space is what are some of the ways that we might see AI crime becoming organised, what are some of the forms of organised cyber criminality that we might see with this, how AI capability is going to be commodified? How are they going to be packaged and sold and distributed in communities? So that's what really is the basis of the next few things that I'll talk about. To let you in a little bit into this space, probably the most emblematic object is the rise of these illicit chatbots and dark LLMs.

They are widely claimed, if you look across any cyber security blog, to be these unrestricted versions of ChatGPT that can help you write code or it can help you put together a phishing email. We found quite a lot of these tools in our research that are definitely called things like FraudGPT, WormGPT, EvilGPT, FreedomGPT. I think they just press [inaudible] like a button or something like that. When we look at some of the advertisements for these tools, we see some pretty consistent claims that these are models that have no guardrails.

You can ask it any question and it won't refuse your prompts. We see explicit marketing for criminal utility. So for fraud, for phishing, for malware, for social engineering and different sorts of access through either subscription models, through trials, through Telegram channels, support websites to accommodate. This is a little bit about what these [inaudible] tools and their different forms look like. So this is across both clear web markets, or surface web environments, Telegram applications and dark web forums as well. You can just see the "welcome to FraudGPT, unrestricted AI without limitations".

And "hey Andrew, welcome to EvilGPT, the most powerful, uncensored AI assistant". You can see some of the marketing claims that are going around for some of these tools, as you might be able to pick up from the title. Although I'm a bit of a sceptic about these claims that are being made in cyber security blogs and in some other circles as well, that there is this really mature illicit market here for these unrestricted chatbots. For the most part, they're really visible, but the ecosystem itself is full of just clones and repeated branding.

So there's like 5 or 6 different FraudGPT. You can see when we actually look at what people are saying in the communities, like, this seller is the only seller of the real GhostGPT, so definitely send them your money. So it's this unexpected secondary scam that seems to be happening where people are interested in accessing these tools, but then they get scammed paying for subscriptions for some of these. So there's a lot of scepticism in cybercrime communities and markets that this is actually a legitimate market space to get involved in, and really unclear technical capability.

there's no way to be able to verify some of the claims that are being made from many of these models. And I'm not saying that [inaudible] that these models don't exist because they do in different capacities. But there's questions too about the technical capability of these models. And are they actually trained on dark web material, or is it just a wrapper of another AI model that might be available? When they do exist, we see different forms of commodification. So this is one that exists in a Tor browser-based service.

It doesn't generate money through subscriptions. It generates money by advertising out to dark web crypto markets. So we see some forms of commodification occurring through this. But it's not exactly in a way that [inaudible] that we think it's taking place. One of the key things that I suppose I want to emphasise, in popping the balloon and disappointing people with the fact that many of these FraudGPT, EvilGPT type services might be scams, is that I actually think that's a helpful finding for us, because it helps us direct our attention to what's actually taking place in organised cybercrime with AI — what are the ways that cybercrime communities or online illicit markets are integrating AI tools?

And as we know, prevention is so often shaped by, I think, how we actually define the problem that we're looking at. And I think a consequence of that is that we've been almost chasing these dark LLMs and these unrestricted chatbots, and the detriment of that has been overlooking these other ways that organised crime actors are actually integrating AI into their products and services. So when we look at what is actually taking place in cybercrime communities, we see in way more situations than not actually quite a broad spectrum of tools being mentioned.

Most often, there is no need to go and try and find these ScamGPT or FraudGPT type services because people just get recommended local or offline unrestricted models that are available through GitHub repositories or through Hugging Face as well. And so there's somewhat of a nice security-efficiency trade-off that's taking place here. There's no need for cybercrime actors to be able to access these tools. And many of these uncensored and unrestricted models are already available. They may not be necessarily marketed for fraud or for scams, and they may require more compute on the user end than a simple Telegram-based messaging app chatbot service.

But it's not the most. What seems to be taking place in these communities, the other things around these loose forms of social organisations. So sharing tips and prompts for how to jailbreak commercial models, we're seeing that increasingly commodified as well. So paid access into, let's say, a Discord server where you can pay $6 and you then have access to all of the fancy jailbreak [inaudible] methods that people are co-developing in amongst their communities, and also the way that AI is now being used as an infrastructure to support existing online markets.

We know, when we look at [inaudible] the sale of drugs through social media, for example, many offenders now are just opting to tag off the back of trending hashtags to promote their illicit drugs or other services. So rather than trying to do [inaudible] like hashtag Weed Brisbane, they would just do a trending hashtag already and try and siphon off some of that traffic. We know, for example, too that AI is being used in things like cryptocurrency trading bots, for example, for offenders to money launder. They can use AI bots to do some of that and the rise of new markets as well.

So something unanticipated has been the rise of secondary markets. So the resale of stolen ChatGPT or other AI accounts from people, the supply of API keys for AI systems as well, they are actually the markets that are happening as opposed to some of these like FraudGPT, ScamGPT type models. So that kind of leads me to linking this back to this idea of AI safety and what it is that looking at some of these organised cybercrime communities can help us all with this goal of a safer AI for all of us to enjoy.

I think as much as in criminology, usually the disciplines I'm from, we like to look at some of the downstream consequences of things, and we try and come up with measures to counteract that. We know that people are getting scammed through ChatGPT, so we can try and come up with countermeasures to help that. A lot of organised crime research argues that we should be looking upstream to the criminal networks that might be facilitating the large scale and the democratisation of AI capabilities. And there is clearly a strong demand for AI — I don't want to get that mistaken as well.

There's clearly a really strong demand for dark AI in cybercrime communities, but there's not yet the supply of that. So I think as soon as [inaudible] a few problems are solved and the supply and demand, then we might be able to see this rise in this market. So that's all for me. I think it's time for questions. Probably squeeze in. Thank you.

Greg Sadler

So maybe kind of a taxonomy question is the most helpful thing here. We talked a bit in the opening session about these voice cloning scams. And that's just a publicly available tool. Anyone can go on the internet and pay a few cents for a voice clone. And then there was a well-publicised example where Anthropic Claude was used to conduct cyber espionage, but that was just done directly through their API. And then there's these tools that are on the dark web. So are those the three buckets of tools that are getting misused here,

Andrew Childs

or is there more to that taxonomy? Yeah, I suppose that would be right. But I would probably say that they're probably interlinked in some way. What we're seeing in cybercrime communities is that people will be discussing and anticipating, or they are anticipating, developments in AI and trying to take a look at what's happening with these Mythos examples and trying to integrate that into their offending. Yeah.

Greg Sadler

Okay, that's probably all the time that we have. So thank you for being here to present. And thank you everyone for coming. We'll have a few minutes to change over rooms before we launch into our next talk. So thank you.