Civitai: Cloud Provider Cuts Ties with Controversial AI Image Platform Amid CSAM Allegations

OctoML has severed its business relationship with Civitai following a 404 Media investigation revealing that Civitai's text-to-image platform was used to generate images that could be classified as child pornography. Civitai previously utilized OctoML's OctoAI for its image generation services. Initially, OctoML planned to continue its partnership and introduced new measures to limit harmful image creation. However, recent reports prompted a complete withdrawal.

Internal communications indicated that OctoML was aware that some Civitai users were producing sexually explicit content, including nonconsensual images of real people and pornographic depictions of children. Although OctoML implemented a filter to block all NSFW content on Civitai prior to its exit, the decision to cut ties was made shortly thereafter.

In response to the investigation, Civitai introduced new moderation methods, including a filter called Civitai Safe Helper (Minor), designed to prevent the generation of images of children if mature themes or keywords are detected. This move followed criticism of Civitai’s “bounties” feature, which incentivized users to create realistic images of real individuals, leading to nonconsensual deepfakes targeting celebrities and private citizens, primarily women. Although Civitai added a filter to limit such content featuring “certain” celebrities, concerns about the platform's safety and ethics persisted.

OctoML, leveraging Amazon Web Services for its operations, expressed in a statement, “We have decided to terminate our business relationship with Civitai. This decision aligns with our commitment to ensuring the safe and responsible use of AI.”