'Configuration Issue' Allows Users AI Generate Nonconsensual Videos

Hey there,

I've written about Civitai a lot since we launched 404 Media, but this is one of the worst things I've ever seen the site do. I expect that Civitai will fix this issue eventually one way or another, but the bigger problem here is AI "image-to-video" generators. As far as I'm aware, there is no good way to moderate them as long as they allow for adult content, and they are getting more popular on the internet than most people realize.

-Emanuel

Civitai, an AI model sharing site backed by Andreessen Horowitz (a16z), is allowing users to AI generate nonconsensual porn of real people, despite the site’s policies against this type of content, increased moderation efforts, and threats from payment processors to deny Civitai service. 

After I reached out for comment about this issue, Civitai told me it fixed the site’s moderation “configuration issue” that allowed users to do this. After Civitai said it fixed this issue, its AI video generator no longer created noncosnensual videos of celebrities, but at the time of writing it is still allowing people to generate nonconsensual videos of non-celebrities.

404 Media saw people actively using Civitai AI video generation to create nonconsensual content, highlighting an inherent difficulty with Civitai and any generative AI tool that allows creating adult content: It is difficult if not impossible to prevent people from abusing those tools. 

In addition to allowing users to share AI image and video generation models that have been customized to render the likeness of real people or highly specific sex acts, Civitai also has an on-site AI image and video generator. Like many other generative AI tools, it allows users to type text prompts in order to produce images or videos. Civitai’s on-site generator can also use the many custom models the site hosts in order to generate media. 

Earlier in May, I reported about a Telegram bot that allowed users to generate nonconsensual short porn videos of anyone for a small price. Shortly after I published the story, Telegram banned the bot. After the bot was banned, one Telegram community dedicated to sharing nonconsensual AI generated porn started sharing other methods for generating the same type of videos without the bot. One user shared instructions for how to generate those types of videos, and other nonconsensual porn videos, on Civitai. 

“With 20 bucks i was able to make 30 5 second videos,” that user said on Telegram. 

I followed the simple instructions that user provided and was able to easily generate porn videos of anyone I had an image of. After providing an image, Civitai allows users to add “Additional Resources,” meaning custom AI models hosted on the site. When a user clicks this option, Civitai brings up a menu of AI models to choose from. The majority of the top models the user sees first are explicitly designed for creating porn. Some of these top models have titles like “Wan Cumshot,” “Wan POV Missionary,” “Wan Cowgirl,” and “Wan POV Blowjob.” Wan refers to the open-weights Alibaba developed AI model it’s based on. If the Civitai user changed their settings to allow “NSFW” content, these models show up first. 

To generate videos, Civitai users have to spend “Buzz,” the site's on-site currency that they can buy or earn by performing certain actions, like completing “Bounties.” Users can buy Buzz in bundles ranging from 5,000 Buzz for $5 to 40,000 Buzz for $40. The price of generations varies based on how long a video is and what “Additional Resources” it’s using, but all the generations I paid for cost between 400 to 600 Buzz. 

In April, Civitai introduced new rules and moderation efforts against nonconsensual content as well as a few other types of adult content after payment processors threatened to stop processing payments for Civitai unless it makes these changes. At the time, Civitai announced that it’s partnering with “Clavata, whose image ingestion and analysis system represents the most advanced and accurate solution we have encountered to-date. This new system will work in tandem with our in-house scanning tools to significantly improve the reliability of image tagging and rating.”

“Every video request now passes through two layers of review,” Civitai’s CEO Justin Maier told me in an email on Monday. “1. Pre-generation gate – If the seed image does not contain verifiable AI-generation metadata, the request is limited to a content level that prohibits sexual content and other mature themes. 2. Post-generation scan – Each video is then frame-scanned by Clavata under a custom policy that applies a content level (PG–XXX) and blocks anything that violates our Terms of Service, including CSAM, incest, or non-consensual likeness.”

Maier said that “The post-generation video scan was not firing as expected for a subset of requests. We identified and fixed the configuration issue within hours of your note.”

Maier did not immediately respond to a follow up question about why the site is still allowing users to generate nonconsensual content of random people and how the company plans to prevent this in the future. 

In 2023, I reported that Civitai was generating images that “could be categorized as child pornography,” according to OctoML, the company that was helping it generate images on the backend. After 404 Media published that story, OctoML dropped Civitai as a client, and Civitai introduced new guardrails to prevent people from generating nonconsensual content on-site. One basic measure Civitai introduced made it so its on-site AI image generator would refuse to generate prompts that described celebrities in sexual scenarios. After this change, for example, the image generator refused to generate an image of “Taylor Swift nude.”

Our previous reporting showed that “image-to-video” tools are currently harder to moderate. It’s much easier to filter out text like “Taylor Swift” and refuse to generate images that include that phrase in a written prompt than it is to detect whether the likeness of a celebrity is included in an image a user is trying to animate. Additionally, there’s no practical way I’m aware of to filter out the likeness of non-celebrities. The only surefire way of preventing people from generating nonconsensual sexual videos on a platform like Civitai is preventing them from generating any type of adult content, period. 

In 2023, 404 Media first reported that Civitai had raised $5.1 million in a seed funding round led by a16z. The investment is notable not only because Civitai took money from one of the most influential venture capital firms in Silicon Valley, but because according to a 2024 transparency report, Maier said that while revenue is growing, the company is still “operating at a loss.”

According to that report, with the exception of salaries, most of the site’s expenses are in “GPUs: Providing generation, training, and content labeling services.” In other words, the company spent $1.2 million, or 22.40 percent of its total spend, mostly on generating media. Most of the revenue came from generation as well. $1.7 million or 80.31 percent of its revenue came from “onsite image and video inference using Yellow Buzz.”

“We haven't had any more rounds of investment since our Seed round with a16z,” Maier told me. “Civitai is currently self-sustaining on product revenue.”