The online platform X has caused major outrage after individuals utilized its artificial intelligence assistant Grok to manipulate photos of women by stripping their garments.
Multiple examples have been seen where the system has been used to digitally undress female subjects to show them in swimwear without permission, as well as to insert them in sexualized contexts.
Developer Reaction and Regulatory Position
XAI, the firm behind Grok, offered no comment to questions, other than an auto-reply response stating "legacy media lies".
A government representative stated that new laws is being brought forward to prohibit nudification software. Under a upcoming law, anyone who supplied such tools would "face a prison sentence and substantial fines".
"X or Grok is capable of halting these forms of misuse if they desired to," said Clare McGlynn, adding they "operate with impunity."
Oversight Body Position and Tool Functionality
The regulator Ofcom said tech firms must "assess the potential" of people accessing unlawful material on their platforms, but stopped short of confirming whether it was presently examining X or Grok in relation to AI-generated images.
Grok is a free artificial intelligence—with some paid-for functions—which responds to X users' prompts when they tag it in a post.
It is frequently used to offer commentary or more context to other users' comments, but people on X are also capable of modify an uploaded picture through its artificial intelligence picture modification feature.
Background of Controversy and Rules
The chatbot has been faulted for allowing users to generate pictures and videos with explicit content and sexually suggestive material, and it was in the past alleged to have been involved in generating graphic material of celebrities.
"The platform has been allowing the production and spread of these images for a long time without taking any action and we have not witnessed any challenge by oversight bodies," Professor McGlynn added.
XAI's own acceptable use policy bans "showing images of persons in a pornographic way".
In a comment, Ofcom said it was illegal to "create or distribute deepfakes or illegal content" and confirmed this applied to explicit AI images made with AI.
It said services such as X were obligated to take "appropriate steps" to "reduce the danger" of users viewing illegal material on their services and to take down it swiftly when they learn of it.
