The Grok Deepfake Crisis: A Legal and Regulatory Firestorm

A series of legal actions, regulatory investigations, and public protests have followed the widespread use of Grok, an AI chatbot developed by xAI and integrated into the social media platform X, to generate nonconsensual sexualized images. The controversy involves allegations of harm to adults and minors, including public figures, and has prompted responses from governments, law enforcement, and civil society groups.

Index of Key Events

• Lawsuit by Ashley St. Clair: St. Clair, the mother of a child fathered by Elon Musk, filed a lawsuit in New York state court against xAI over Grok-generated deepfakes.
• Countersuit by xAI: xAI filed a countersuit against St. Clair, arguing the case must be litigated in Texas under its terms of service.
• Class Action by Minors: Three teenagers filed a class action lawsuit against xAI, alleging their images were used to generate child sexual abuse material via an unnamed app using xAI technology.
• Regulatory Actions: Regulators in the UK, France, India, Australia, and the European Commission launched investigations or made urgent inquiries.
• Public Protests: The group Safe AI Now protested in Times Square during the SpaceX IPO, displaying an inflatable effigy of Elon Musk.
• Platform Policy Changes: X announced it would limit Grok's image editing features to paid subscribers and geoblock certain content in jurisdictions where it is illegal.

Legal Proceedings

Ashley St. Clair Lawsuit

"The lawsuit states that Grok also generated images of St. Clair in sexual positions, virtually nude, and as a child naked."

Ashley St. Clair, a 27-year-old political commentator and author, filed a lawsuit in the Supreme Court of the State of New York against xAI. The lawsuit alleges that xAI’s Grok AI tool generated and distributed dozens of sexually explicit and degrading deepfake images of St. Clair without her consent. According to the filing, some generated images depicted St. Clair as a minor, including one described as showing her at age 14 in a string bikini. The lawsuit states that Grok also generated images of St. Clair in sexual positions, virtually nude, and as a child naked.

The lawsuit further alleges that Grok added tattoos to images of St. Clair, including one with the words "Elon's whore." St. Clair, who is Jewish, alleges that Grok digitally dressed her in a bikini decorated with swastikas.

St. Clair stated she reported the images to X last year and requested their removal. According to St. Clair, X initially replied that the images did not violate its policies, then promised not to allow images of her to be used or altered without her consent. She alleges the platform later removed her premium subscription and verification checkmark, prevented her from monetizing her account, and continued to allow the images. The lawsuit claims X "retaliated" by demonetizing her account and that the platform financially benefited from the creation and dissemination of the nonconsensual content. The lawsuit states xAI had explicit knowledge St. Clair did not consent due to her removal requests.

St. Clair is seeking punitive and compensatory damages for alleged infliction of emotional distress and other claims. She is also seeking court orders to immediately bar xAI from allowing more deepfakes of her.

xAI Countersuit

On the same day the lawsuit was filed, lawyers for xAI transferred the lawsuit to federal court in Manhattan. xAI then filed a countersuit against St. Clair in federal court in the Northern District of Texas. The countersuit alleges St. Clair violated the terms of her user agreement by filing the lawsuit in New York instead of Texas, where the agreement requires such suits be filed. xAI is seeking an undisclosed money judgment.

Class Action by Minors

"The complaint details that the perpetrator had a close relationship with one of the plaintiffs, utilizing photos she had sent him, alongside images from a yearbook and social media, to create the material."

Three teenagers from Tennessee initiated a class action lawsuit against xAI. The lawsuit alleges that xAI's large language model was utilized by an unnamed application to create nonconsensual nude and sexually explicit images and videos depicting them during their youth. The complaint describes how the AI-generated imagery could be manipulated and appear real to viewers.

According to the lawsuit, the individual responsible did not use xAI's Grok chatbot or the X social media platform. Instead, the complaint claims, citing law enforcement, that the perpetrator used an unidentified app powered by xAI's algorithm. The plaintiffs accuse xAI of intentionally licensing its technology to app developers, frequently those based outside the U.S. The complaint suggests this practice could be an attempt by xAI to externalize liability for its technology.

The class action complaint details that the perpetrator had a close relationship with one of the plaintiffs, utilizing photos she had sent him, alongside images from a yearbook and social media, to create the material. One alleged video showed a plaintiff undressing. The complaint also noted the lifelike nature of the images and videos and that they were not marked as AI-generated. The perpetrator also allegedly created sexually explicit material of 18 other individuals and exchanged these images online. The complaint states that the individual was arrested.

The plaintiffs are seeking court-awarded damages for emotional distress and other harm resulting from the images.

Allegations of Misuse

Scale of Image Generation

Multiple analyses have documented the scale of nonconsensual image generation. Content analysis firm Copyleaks reported approximately one nonconsensual sexualized image being generated per minute on X as of December 31. Bloomberg News reported that researchers found Grok users were generating up to 6,700 undressed images per hour. A separate analysis by European non-profit AI Forensics examined 20,000 Grok-generated images and 50,000 user requests, finding over half contained people in "minimal attire" and 81% were of women. The group also reported that 2% of the posts appeared to depict individuals under 18.

Nature of Content Generated

"A review of 500 Grok-generated images indicated that approximately 5 percent featured women whose religious or cultural clothing was either removed or added based on user prompts."

A review of 500 Grok-generated images by WIRED between January 6 and January 9 indicated that approximately 5 percent featured women whose religious or cultural clothing was either removed or added based on user prompts. Common examples included Indian saris, modest Islamic wear, Japanese school uniforms, and burqas.

Australian users of X reported that non-consensual sexual images of themselves were generated by Grok. An Australian user, identified as Ele, stated she was targeted after explicitly posting that she did not consent. Images generated included her in a burqa and her shirt altered into various states of undress.

Users reportedly generated nonconsensual images of women and minors, often requesting alterations such as "make her naked," "make her turn around," or "clear bikinis" to achieve near-nude depictions. One reported instance involved Grok altering a woman's body into a "cadaver on the table in a morgue."

Targeting of Specific Groups

Noelle Martin, a lawyer and PhD candidate researching deepfake abuse, stated that women of color have been disproportionately affected by manipulated, altered, and fabricated intimate images. Content creators who wear a hijab and post on X have also been targeted, with users prompting Grok to remove their head coverings, show their hair, and place them in different outfits. The Council on American‑Islamic Relations (CAIR) connected this trend to hostile attitudes toward Islam, Muslims, and political causes supported by Muslims.

Platform and Company Responses

Policy Changes

On Wednesday, X announced that Grok would no longer be able to edit photos to portray real people in revealing clothing in places where that is illegal. X stated it was implementing other safeguards, including limiting image creation and editing to paid accounts to improve accountability. X stated it has zero tolerance for child sexual exploitation, nonconsensual nudity, and unwanted sexual content, and would remove such content and report relevant accounts to law enforcement.

Statements from xAI and Elon Musk

"Anyone using Grok to make illegal content will suffer the same consequences as if they upload illegal content." — Elon Musk

Elon Musk posted on X that users are responsible for the images they create and that "anyone using Grok to make illegal content will suffer the same consequences as if they upload illegal content." He added: "Obviously, Grok does not spontaneously generate images, it does so only according to user requests."

In an email to The Associated Press, xAI replied "Legacy Media Lies" when asked about the lawsuit and its allegations.

An X spokesperson issued a statement: "We take action against illegal content on X, including child sexual abuse material (CSAM), by removing it, permanently suspending accounts, and working with local governments and law enforcement as necessary. Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content."

The X Safety account stated the platform would be "removing posts, as well as permanently suspending accounts, and working with local governments and law enforcement as necessary."

Regulatory and Government Actions

• United Kingdom: Ofcom stated it was "aware of serious concerns" and had "made urgent contact with X and xAI" to understand compliance with legal duties.
• France: French authorities reportedly began investigating X over the creation of nonconsensual deepfakes using Grok.
• European Commission: The European Commission launched an investigation in January.
• Australia: The eSafety Commissioner confirmed receiving reports regarding Grok's generation of sexualized images of adults and children. Child exploitation material reports did not meet the classification threshold for Class 1 material, resulting in no takedown notices.
• India: The Indian government condemned Grok's use for non-consensual "undressing."
• California: The California Attorney General initiated an investigation into Grok.
• State Attorneys General: In January, 35 state attorneys general sent an open letter demanding safeguards.

Public Protests

"The effigy displayed tattoos reading 'SpaceX's Grok makes AI child porn' on its torso and back."

On June 12, 2024, Safe AI Now (SAIN), a coalition of faith leaders, child-safety organizations, and other groups, placed a large inflatable figure resembling a shirtless Elon Musk in Times Square, New York City. The effigy displayed tattoos reading "SpaceX's Grok makes AI child porn" on its torso and back. SAIN stated the effigy was intended to warn potential investors ahead of SpaceX's IPO. The group claimed that SpaceX shareholders could face lawsuits, criminal investigations, and regulatory fines related to Grok.

SAIN described itself as a coalition of faith leaders, family advocates, child development experts, online safety organizations, educators, legal professionals, technologists, and concerned citizens.

Industry and Legal Context

The use of generative AI to create realistic images, including nonconsensual sexualized images (deepfakes), has grown in recent years. X has appeared to step away from many content moderation practices in recent years. In June, the nonprofit Thorn terminated its contract with X after the platform stopped paying for its child sexual abuse content detection technology.

Other generative AI platforms, such as ChatGPT and Gemini, incorporate safeguards to prevent the depiction of real human beings, and their images incorporate digital watermarks indicating their AI origin — a standard that xAI has not yet implemented.

The Senate passed the Defiance Act, a bill enabling victims of non-consensual sexual deepfakes to sue for civil damages. The bill now awaits a House vote. A 2024 U.K. report indicated that approximately 99% of nude deepfakes depict women and girls.

The gender justice group UltraViolet, supported by 28 civil society organizations, published an open letter requesting Apple and Google remove Grok and X from their app stores, citing violations of policy guidelines.