Graphic by Max Weinstein
When Syracuse University sophomore Rachel Feuchtwanger saw Instagram posts in early September claiming a court had legalized child sexual abuse material (CSAM) generated by artificial intelligence (AI), the story brought her back to high school, when one of her classmates created AI-altered sexual images of girls in his grade using photos found on their Instagrams. The boy was suspended for a few days, she said, but nothing else happened. There was no law in place to address what he had done.
“I remember them feeling like their reputations were ruined, and girls were crying in the hallways,” Feuchtwanger said. “If states are able to legalize this, and it becomes okay for people to make and have these images, it’s going to be very weird and dangerous, especially for minors.”
In August, the 7th U.S. Circuit Court of Appeals threw out one of four federal charges against Steven Anderegg, a Wisconsin man accused of using AI to create sexually explicit images of children and posting them on Instagram. The dismissed charge was for possessing the images in his home. The other three charges, which include making and sharing the images and sending them to a minor, are still pending. Anderegg has not yet been convicted of any crime.
Many people assumed the government had publicly deemed the sexual exploitation of minors practically legal with this ruling. Experts say the reasoning behind the court decision is easier to understand if you set aside the images and think about the government’s power.
“‘Nudified’ images of real children are not First Amendment protected because they do real harm to the real children depicted, and multiple courts have said so,” Riana Pfefferkorn, a policy fellow at Stanford University’s Institute for Human-Centered AI who studies the impact of AI-generated CSAM, wrote in a statement to Jerk Magazine.
The ruling concerned images of children who don’t exist, unlike the victims from Feuchtwanger’s high school. Its scope was also far narrower than the posts suggested.
The 7th Circuit Court’s decision on the Anderegg case rested on Stanley v. Georgia (1969), a Supreme Court precedent that holds the government can’t prosecute someone for privately owning obscene material inside their own home. If courts let the government decide what people can keep in their homes, Pfefferkorn said, that power wouldn’t stay limited to material nearly everyone finds repulsive.
Pfefferkorn said the definition of “obscenity” is ever-changing because things considered “obscene” shift with cultural norms.
“After all, some state and federal government officials have a very expansive definition lately of what they consider ‘obscene’—such as legal adult pornography, and indeed the very existence of queer and trans people at all,” Pfefferkorn wrote. “Without a strong First Amendment, the government would get to reach into your home and decide what books and magazines you’re allowed to possess, what art you’re allowed to have on the walls, etc.”
That’s why, Pfefferkorn said, cases like this one matter—even when the material is widely considered disturbing.
Under the First Amendment, speech always begins protected. The government has to clear a very high bar to take that protection away. When examining cases attempting to censor speech, courts apply a “strict scrutiny” test. This requires the government to prove two things: that the attempt to censor speech serves a compelling government interest, and that the law is written as narrowly as possible to serve that interest. Protecting children from sexual exploitation would normally clear that bar easily. But here, the children in the images don’t exist.
“One of the things we always want to look for in law is the harm that’s happening,” said Alexis Ingber, assistant professor of communications law at the S.I. Newhouse School of Public Communications. “It seems very obvious that there are a bunch of harms with having child pornography, because having it means that it was created. If you create something that’s obscene, but it’s all fake, who is getting harmed in that situation? I don’t know.”
Ingber’s internal conflict is shared by many legal experts, including Roy Gutterman, director of Newhouse’s Tully Center for Free Speech.
“As distasteful and disturbing as this content is, it is hard to criminalize the content without real, live victims,” Gutterman wrote in a statement to Jerk.
Gutterman emphasized the ruling of Ashcroft v. Free Speech Coalition (2002), another case that stood as precedent in the recent decision.
In that case, the Supreme Court ruled that content starring adults who appeared to be minors was protected by the First Amendment, declaring virtual CSAM lacks a direct, non-contingent link to real-world child abuse. Essentially, just because a piece of media has similarities to illegal content, it is not illegal itself.
Because AI is a relatively new technology, there has been no case law about its regulation. And because the Supreme Court hasn’t yet issued a ruling, judges have to rely on old case law to make present decisions.
Critics of the ruling argue that the images aren’t victimless.
“AI is still getting its images from somewhere,” Feuchtwanger said. “It’s pulling from the internet.”
New York State Sen. Patricia Fahy (D) has proposed legislation to criminalize possession of AI-generated CSAM. Her office, like Feuchtwanger, argues that AI is trained on the likeness of real children that were thus harmed in the process.
While it’s understandable why this would raise concerns, Pfefferkorn explained that it can be hard to tell whether an image generator is basing its content on real sexually explicit content involving minors. In many cases, these platforms gather visual data from both normal photos of children and legal adult pornography, and combine the two.
“An image generator knows what goggles look like and knows what a bunny looks like, and so it can produce a picture if you prompt it for an image of a bunny wearing goggles,” Pfefferkorn said.
Given this complexity, Ingber does not expect Fahy’s legislation, or any similar to it, to succeed. If a court were to rule that such a law is constitutional, Ingber said, it would create a slippery slope that could ultimately stop AI development as a whole. If a million people are used as training material for an AI that ends up producing harmful content, it would currently be almost impossible for any one of those individuals to claim direct harm.
Under the proposed legislation, however, the company could be liable to everyone—regardless of how much harm they were actually caused.
“You can imagine how a lot of AI companies would not want that to be the case, because then anything they do is based on real people.” Ingber said. “They’ll argue that none of the training data is identifiable, that it’s all been disaggregated. And it’s a blend of so many people that you can’t even say.”
Government regulation of AI companies in general is difficult to solidify at present.
“The biggest issue is the ‘who,’” Gutterman wrote. “Who is responsible or liable for AI generated content whether it somehow defames someone or infringes on intellectual property rights or causes some other harm?”
The answer: We aren’t sure yet.
But when asked what the future could look like for AI regulations in the United States, Ingber did not have high hopes.
“Do I think there’ll ever be a federal AI law about what platforms are allowed to have? No, not anytime soon. Maybe there will be state law, but it would again be subject to First Amendment strict scrutiny. It would have to be so limited,” Ingber said.
At its core, this case brings up a pressing legal issue that involves something close to all of us: our data. To Ingber, the case raises more red flags regarding the lack of regulations on data privacy.
Much of the information AI uses comes from content everyday people decide to share online, which has yet to receive any federal protection from being distributed to companies for this exact purpose.
“We don’t even have data privacy law in the U.S.,” Ingber said. “So I can’t optimistically say that I think there’s going to be some amazing AI law to protect people.”