Apple and Google have been ordered to remove apps that can “nude” or “undress” people and told they must stop profiting from the harmful technology, according to cease-and-desist letters sent to the companies by WIRED.
On Thursday, San Francisco City Attorney David Chiu sent legal notices to Apple and Google, asking them to remove from their app stores 13 face-swapping apps that allow users to create AI-generated, non-consensual nude images. The letters say that the Silicon Valley giants should stop “supporting” the sale of explicitly fake images and “cut off” business relationships with the app developers.
“Generating non-consensual intimate images is illegal, harmful and completely unacceptable,” Chiu tells WIRED. The city attorney, whose office has previously taken legal action against 16 popular deepfake websites, says Apple and Google have likely earned “millions of dollars in fees” from apps that offer nudification and that they should improve their moderation processes to prevent them from appearing in their stores in the first place.
“These companies have a responsibility to ensure that apps on their platforms do not facilitate sexual abuse,” says Chiu. The city’s legal brief states that California law prohibits supporting services that create deepfake pornography. The apps use in-app payments, from which the tech companies receive a cut, the letters say. “The fact that some of the world’s largest and most established technology companies are enabling this must stop.”
Researchers have repeatedly found and reported apps on Apple’s App Store and Google’s Play Store that allow people to use AI to generate sexual images – including some apps that have been deemed suitable for children. As new laws and bans aim to combat the scourge of explicit deepfakes online, technology and social media companies continue to alert millions of people to the harmful technology.
Both Apple and Google have developer policies that prohibit pornography, abuse, and harassment on their platforms. According to reports from researchers and journalists, they have previously removed dozens of Nudify and deepfake apps.
Google spokesman Dan Jackson tells WIRED that the company has removed “hundreds” of apps with policy violation alerts, including the five Android apps flagged by Chiu’s office, and has taken other steps to restrict access to them.
“Google Play does not allow apps with sexual content, and we continually take proactive steps to identify and remove apps with harmful content,” Jackson said in a statement. “When violations are reported to us, we investigate the matter and take immediate action. In the case of these apps, this included blocking hundreds of violating apps and restricting related search terms such as ‘nudify’ from our store.”
Apple tells WIRED that developers are ultimately responsible for the content of their apps and that those who don’t follow the rules will have their apps removed. The company also says that users can self-report potentially problematic apps. Adam Dema, an Apple spokesman, told WIRED in a statement that the company has removed three of the apps flagged by the San Francisco district attorney and is “in the process of terminating their developer accounts from our program.” The other four developers must “fix policy violations or risk being removed as well,” the company says.
“The App Store was designed to be a safe and trusted place for users, and we have always strictly banned apps that aim to generate, distribute or consume pornography,” Dema said in the statement. “‘Nudification’ apps violate our app review policies and we have proactively rejected many of these apps and removed many others, even when users have reported them through our reporting tools.”
https://www.wired.com/story/san-francisco-demands-apple-and-google-delete-ai-nudify-apps-from-app-stores/
