Muah.AI is a website where you can create an AI girlfriend, a chatbot that you can talk to via text or voice and send images of yourself upon request. The service has nearly 2 million registered users and its technology is said to be “uncensored.” And judging by the data allegedly stolen from the site, people may be using the tool in an attempt to create child sexual abuse materials (CSAM).
Last week, 404 Media’s Joseph Cox first reported on the dataset after being alerted to its existence by an anonymous hacker. What Cox discovered was very alarming. Cox identified one prompt that included language about orgies involving “new-born babies” and “young children.” This indicates that users asked Muah.AI to respond to such scenarios, but it is unclear whether the program did so. Major AI platforms such as ChatGPT employ filters and other moderation tools aimed at blocking content generation in response to such prompts, but lesser-known services have issues. There tends to be less.
People have been using AI software to generate sexually exploitative images of real individuals. Earlier this year, a porn deepfake of Taylor Swift was circulated on X and Facebook. And child safety advocates have repeatedly warned that generative AI is now being widely used to create images of real-life child sexual abuse, an issue that has surfaced in schools across the country.
The Muah.AI hack is one of the clearest and most public examples of this widespread problem to date. Perhaps for the first time, the scale of the problem is set out in very clear terms.
I spoke to Troy Hunt, a prominent security consultant and creator of the data breach tracking site HaveIBeenPwned.com, after seeing a thread he posted on X about the hack. Mr. Hunt also sent Muah.AI data from anonymous sources. After reviewing the data, we found numerous instances where users were prompting the program to provide child sexual abuse material. A search for data on 13-year-olds yielded more than 30,000 results, “many alongside prompts describing sexual acts.” When he tried it before puberty, he got 26,000 results. He estimates that there are tens, if not hundreds of thousands, of prompts for creating CSAM in the data set.
Hunt was surprised that some Muah.AI users didn’t even try to hide their identities. In one case, he matched an email address obtained from a breach to a LinkedIn profile belonging to an executive at an “ordinary” company. “I looked up his email address and it was literally like his name dot last name at gmail.com,” Hunt told me. “There are many cases where people try to obscure their identities, but if you can pull the right strings, you can find out who the person is. But this guy didn’t even try. Hunt said CSAM is traditionally associated with the fringes of the Internet. “The fact that this is on a mainstream website probably surprised me a little more.”
Last Friday, I contacted Muah.AI and asked about this hack. A person named Harvard Han, who runs the company’s Discord server, confirmed to me that his website had been compromised by hackers. I asked him about Hunt’s estimate that there could be hundreds of thousands of prompts in the dataset for creating CSAM. “That’s not possible,” he told me. “How is that possible? Think about it. There are 2 million users. There’s no way that 5 percent are pedophiles.” )
When I asked Mr. Hunt if the data he had was real, his first response was: I’m not denying it. ‘But later in the same conversation, he said he wasn’t sure. Mr. Han was traveling, but said his team would investigate.
Han repeatedly emphasized that the site has a small staff and limited resources to monitor user behavior. There are less than five people working there, he told me. However, the site seems to have built a decent user base. According to data provided to me by traffic analysis firm Similarweb, Muah.AI has averaged 1.2 million visits per month over the past year or so.
Last year, Han told me, his team introduced a filtering system that automatically blocks accounts that use certain words in their prompts, such as teenager or child. But he told me that users complained that they were being unfairly banned. He said the site then adjusted its filters to stop automatically blocking accounts, but still prevented images from being generated based on those keywords.
But at the same time, Han told me that his team doesn’t check to see if his company is generating images of child sexual abuse for its users. He said he expects many such requests “will probably be denied, denied, denied.” However, Han acknowledged that savvy users would likely be able to find a way around the filter.
He also offered some sort of justification for why users would generate images depicting children in the first place. Some Muah.AI users grieving the death of a family member are using the service to create AI versions of their deceased loved ones. When I pointed out to Mr. Hunt, a cybersecurity consultant, that he had seen the word 13 years old used alongside sexually explicit acts, Mr. Hunt said, “The problem is, check all the prompts.” We don’t have the resources to do so.” (After Cox’s article about Muah.AI, the company said in a Discord post that it would be experimenting with new automated methods for banning users.)
In short, even the people running Muah.AI don’t know what their service does. At one point, Han suggested that Hunt might know more about the contents of the data set than he did. The fact that a site like this can operate with so little regard for the harm it causes raises the larger question of whether it should even exist at all, when the potential for abuse is so high.
Meanwhile, Han has taken a familiar argument about censorship in the online age and stretched it to its logical extreme. “I’m an American,” he told me. “I believe in free speech. I think America is different. And we believe that AI should not be censored and trained.” And you can use this gun to protect your life, your family, your loved ones, and you can use it to commit mass shootings.”
Federal law prohibits computer-generated images of child pornography that feature real children. In 2002, the Supreme Court ruled that a blanket ban on computer-generated child pornography violates the First Amendment. How exactly existing laws apply to generative AI is an area of active debate. When I asked Han about federal law regarding CSAM, he said Muah.AI only offers AI processing and compared his service to Google. He also reiterated that his company’s word filter may be blocking some images, although he is not sure.
No matter what happens with Muah.AI, these issues are sure to remain. Mr. Hunt told me that he had never even heard of the company until the breach occurred. “And I’m sure there are dozens more.” Muah.AI just happened to have its contents flipped by a data hack. The era where AI will cheaply cause child abuse is just around the corner. What was once hidden in the darkest corners of the internet now seems so easily accessible, but just as alarmingly, so hard to eradicate.
