San Francisco Demands Apple and Google Purge AI ‘Nudify’ Apps From Stores

San Francisco is escalating its fight against AI-generated explicit content by targeting the world’s dominant mobile app marketplaces. City Attorney David Chiu has issued formal cease-and-desist letters to Apple and Google, demanding the immediate removal of 13 applications that utilize artificial intelligence to generate nonconsensual nude images.

The legal warning accuses the tech giants of profiting from and enabling the spread of digital abuse, primarily targeting women and girls. Chiu’s office identified eight offending applications on Apple’s App Store and five on the Google Play Store. While often marketed as harmless face-swapping or photo-editing utilities, investigators discovered these programs are frequently used to superimpose victims’ faces onto sexually explicit material.

By hosting these programs, processing in-app purchases, and collecting commission fees, Apple and Google are effectively “aiding and abetting” the proliferation of deepfake pornography, according to Chiu. The city has given both companies a 28-day deadline to detail how they will comply, cut off payment processing for the developers, and establish permanent screening measures. Failure to act could result in civil enforcement actions yielding penalties starting at $25,000 per violation.

In response to the demands, Google announced it has removed all five flagged Android applications from its platform. The search giant noted it has banned hundreds of similar apps and restricted search terms like “nudify” to curb access. Apple reported that it has removed three of the specified apps and terminated the associated developer accounts, while giving four other developers a strict window to resolve policy violations.

Despite both companies maintaining strict guidelines against hosting sexually explicit material, the presence of these tools has raised serious questions about the efficacy of their automated and manual app review processes.

The scope of the issue extends far beyond these 13 applications. Recent investigations by the Tech Transparency Project identified approximately 100 “nudify” apps across both major digital storefronts. These applications collectively generated an estimated $120 million in revenue and surpassed 480 million downloads.

Furthermore, an academic study conducted by researchers at Cornell University and Georgetown University analyzed 420 face-swap programs available to mobile users. After testing 155 of them, researchers found that 70% allowed users to generate explicit deepfakes without encountering any built-in technical safeguards. Because these tools are often disguised as standard photography software, they easily bypass basic moderation filters.

This ultimatum is part of a broader, multi-front legal campaign against nonconsensual AI imagery. Earlier this year, San Francisco filed lawsuits against 16 prominent AI undressing websites, which had collectively drawn over 200 million visits in the first half of 2024 alone. Social media giant Meta also took legal action against the developers of Crush AI after discovering thousands of ads promoting nudification services across its platforms.

On a federal level, the U.S. Department of Justice recently seized multiple domains linked to websites that published hundreds of thousands of deepfake explicit images depicting high-profile individuals. Chiu emphasized that platform operators must take a proactive stance rather than relying on external complaints to police their marketplaces, warning that further legal actions remain on the table.