Australians are part of the growing number of people worldwide who have been targeted on Elon Musk's AI chatbot Grok, which allowed sexualised images of women and children to be generated without their consent.
Grok Imagine, an AI image generator that allows users to create photos and videos with text prompts, was first launched in July but started causing issues after it was opened to the public in October.
The tool, which is hosted on X, has been allowing users to nudify images of or sexually depict others as recently as today.
READ MORE: ATO vow to ban tax dodgers from leaving Australia
Grok differs from other AI generative platforms as the images are publicly shared and can be easily spread.
Since late last year, the eSafety Commission has received several reports of Grok being used to generate unconsensual sexualised images.
"Some reports relate to images of adults, which are assessed under our Image-Based Abuse scheme, while others relate to potential child sexual exploitation material, which are assessed under our Illegal and Restricted Content scheme," a spokesperson said.
The reports of adults are currently being assessed, while reports of children did not meet the threshold for child sexual exploitation material.
eSafety has not taken any action, but said it was worried about the use of generative AI.
"eSafety remains concerned about the increasing use of generative AI to sexualise or exploit people, particularly where children are involved," the spokesperson said.
READ MORE: AI dominates as Australia's fastest-growing jobs revealed
Analysis from non-profit outfit AI Forensics said in a report that it analysed 20,000 images generated by Grok between December 25 and January 1 and found that two per cent depicted a person who appeared to be 18 or younger, including 30 of young or very young women or girls, in bikinis or transparent clothes.
One of the victims includes Musk's former partner, Ashley St Clair, who became estranged from the tech billionaire after the birth of their child in 2024.
She told The Guardian users have used Grok to create "revenge porn".
"I felt horrified, I felt violated, especially seeing my toddler's backpack in the back of it," she told the publication.
X did not deny that unconsensual sexualised content exists on Grok, but warned that users could be suspended or referred to law enforcement.
"We take action against illegal content on X, including Child Sexual Abuse Material, by removing it, permanently suspending accounts, and working with local governments and law enforcement as necessary," X said through its Safety account.
"Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content."
READ MORE: Aussies among tourists trapped on remote desert island
https://x.com/Safety/status/2007648212421587223
But governments and regulators around the world called for Musk's platform to do more to combat abusive content.
The United Kingdom's Technology Secretary Liz Kendall said the content is "absolutely appalling, and unacceptable in decent society".
"We cannot and will not allow the proliferation of these demeaning and degrading images, which are disproportionately aimed at women and girls," she said.
The country's communications regulator Ofcom said it made "urgent contact" with X.
The European Union said it was aware of the content and strongly condemned Grok.
"This is not spicy. This is illegal. This is appalling. This is disgusting. This is how we see it, and this has no place in Europe. This is not the first time that Grok is generating such output," European Commission spokesman Thomas Regnier said.
France, Poland, India, Malaysia and Brazil also called for Musk to implement strict rules and take action.
– With Associated Press
DOWNLOAD THE 9NEWS APP: Stay across all the latest in breaking news, sport, politics and the weather via our news app and get notifications sent straight to your smartphone. Available on the Apple App Store and Google Play.
