The online platform X has sparked widespread backlash after individuals utilized its artificial intelligence assistant Grok to manipulate photos of women by removing their attire.
Multiple instances have been seen where the technology has been employed to remove clothing from women to render them in swimwear without consent, as well as to insert them in sexually explicit scenarios.
Company Reaction and Legal Stance
XAI, the firm behind Grok, did not respond to requests for comment, other than an automatically-generated response stating "legacy media lies".
A government official stated that legislation is being proposed to outlaw nudification technology. Under a upcoming law, anyone who provided such technology would "face a jail term and substantial fines".
"X or Grok is capable of halting these types of misuse if they desired to," said a law professor, adding they "operate with impunity."
Regulator Position and Platform Functionality
The regulator Ofcom said platforms must "assess the potential" of people seeing unlawful material on their platforms, but stopped short of confirming whether it was at this time looking into X or Grok in concerning artificial intelligence pictures.
Grok is a free artificial intelligence—with some paid-for features—which reacts to X users' requests when they tag it in a message.
It is frequently used to offer reactions or further context to other posters' comments, but people on X are also capable of alter an uploaded image through its AI image editing feature.
History of Controversy and Rules
The tool has been faulted for permitting users to generate images and videos with nudity and sexualised content, and it was in the past alleged to have been involved in making inappropriate material of celebrities.
"The platform has been enabling the production and sharing of these pictures for a long time without taking any steps and we have yet to see any challenge by regulators," the expert added.
XAI's own usage rules bans "portraying likenesses of individuals in a pornographic fashion".
In a comment, Ofcom said it was unlawful to "make or distribute deepfakes or illegal content" and stated this included sexual deepfakes made with AI.
It said platforms such as X were required to take "necessary actions" to "reduce the danger" of users encountering unlawful content on their services and to take down it quickly when they learn of it.
