"The irony," wrote U.S. District Judge Laura M. Provinzino in her order excluding the declaration.
The filing at issue was prepared by Jeff Hancock, co-director of Stanford University's Cyber Policy Center and director of the center's Social Media Lab. Hancock was retained by Minnesota Attorney General Keith Ellison to prepare a brief supporting the state's position in a suit challenging its ban on the dissemination of election-related, AI-generated content.
"Professor Hancock, a credentialed expert on the dangers of AI and misinformation, has fallen victim to the siren call of relying too heavily on AI — in a case that revolves around the dangers of AI, no less," Judge Provinzino said in the order.
Judge Provinzino said she was adding her voice "to a growing number of courts around the country declaring the same message: verify AI-generated content in legal submissions!"
After AI-hallucinated citations in the brief were flagged by the plaintiffs in the case, Hancock offered a "detailed explanation" of his drafting process and how the errors came to be, according to the order. Hancock also assured the court that he "stands by the substantive propositions in his declaration, even those that are supported by fake citations," the judge said.
"But, at the end of the day, even if the errors were an innocent mistake, and even if the propositions are substantively accurate, the fact remains that professor Hancock submitted a declaration made under penalty of perjury with fake citations," Judge Provinzino said.
She added that it's "particularly troubling" that Hancock said he usually validates citations with a reference software when he writes academic articles, but he didn't do so here.
"One would expect that greater attention would be paid to a document submitted under penalty of perjury than academic articles," the judge said. "Indeed, the court would expect greater diligence from attorneys, let alone an expert in AI misinformation at one of the country's most renowned academic institutions."
Ultimately, the judge agreed to exclude Hancock's testimony and also refused to allow Ellison's office to file a corrected version of the declaration. At the same time, she rejected the plaintiffs' request to exclude the testimony of another expert tapped by the attorney general and, in a separate order, shot down their request for a preliminary injunction.
The suit was filed by popular social media personality Christopher Kohls and Minnesota state Rep. Mary Franson and centers on Minnesota Statute Section 609.771, which Kohls and Franson contend is a "flagrant violation of the First Amendment." The statute means criminal penalties are now associated with the dissemination of certain AI-generated political content that could influence an election, according to their complaint.
Franson, a Republican who represents a portion of western Minnesota, said she frequently posts memes to her social media accounts that are often computer-edited or AI-created images or videos that feature "the likeness of real politicians for comedic or satirical effect." The deepfake law means she faces criminal penalties and risks forfeiting her eligibility for political office, according to the complaint.
One of the memes Franson shared was created by Kohls, "who posts humorous content commenting on and satirizing political figures on his popular YouTube, Facebook and X channels using the screenname 'Mr Reagan,'" the complaint states.
Hancock's declaration was filed to support the attorney general office's opposition to the plaintiffs' preliminary injunction bid. In the filing, he said he was retained by the attorney general's office to provide expert opinion and testimony "regarding how AI is influencing misinformation on social media and the psychological impact of deceptive deepfakes, particularly deepfakes shared on social media."
But Franson and Kohls said the declaration cites "an imaginary study" that purports to describe the harmful effects of deepfakes. Specifically, Hancock wrote that a study "found that even when individuals are informed about the existence of deepfakes, they may still struggle to distinguish between real and manipulated content. This challenge is exacerbated on social media platforms, where deepfakes can spread rapidly before they are identified and removed."
That study, according to Hancock, was titled "The Influence of Deepfake Videos on Political Attitudes and Behavior" and posted in the Journal of Information Technology & Politics in 2023. He included a link to the study, but the link doesn't work, Kohls and Franson said.
"Perhaps this was simply a copy-paste error?" they said in their motion to exclude the declaration. "It's not. The article doesn't exist."
In Friday's order, Judge Provinzino said that the office of Attorney General Ellison has since contacted Hancock, who admitted that the declaration "inadvertently included citations to two nonexistent academic articles and incorrectly cited the authors of a third article."
Hancock said he had used GPT-4o, a generative AI tool, to draft the declaration, according to the order.
Judge Provinzino emphasized that she "does not fault" Hancock for using AI to conduct research. In many ways, AI has the potential to "revolutionize legal practice for the better," she said.
"But when attorneys and experts abdicate their independent judgment and critical thinking skills in favor of ready-made, AI-generated answers, the quality of our legal profession and the court's decisional process suffer," Judge Provinzino said.
She noted that Ellison has stated that his office had no idea that the declaration included fake citations. And attorneys for Ellison apologized at oral argument for the citations. Judge Provinzino said she "takes Attorney General Ellison at his word and appreciates his candor in rectifying the issue," she said.
Still, Judge Provinzino "reminded" the attorneys that the Federal Rule of Civil Procedure requires that the "truth and reasonableness" of filings be verified. That duty may now mean that attorneys must ask their witnesses whether they have used AI in drafting declarations and what they have done to verify AI-generated content, she said.
"The court cannot accept false statements — innocent or not — in an expert's declaration submitted under penalty of perjury," she said.
In her separate order on the preliminary injunction, Judge Provinzino said the law at issue does not penalize parody or satire.
"The statute's definition of a deepfake … categorically excludes constitutional parody from its sweep," she said. "If the speech in question 'cannot reasonably be interpreted as stating actual facts about an individual,' then it cannot be a deepfake under the statute."
That means the main question in deciding whether to issue an injunction is if the speech flagged by Kohls and Franson falls "outside of the ambit of parody," according to the order.
Kohls has labeled his videos as parody, meaning he lacks standing to bring a challenge before the law is being enforced against him, Judge Provinzino said. As for Franson, at least some of her speech "falls squarely into the scope of parody and outside the scope of the statute," the judge said. Some of her content sharing is a closer call though, Judge Provinzino said. Ultimately, the judge said Franson does have standing to pursue her claims at this stage.
Still, Franson waited too long to ask for an injunction — and she also voted for the law she is now challenging, the judge said.
"Quite simply, Rep. Franson knew by May 26, 2023, that Minnesota Statute Section 609.771 would be enacted, and that it would bar dissemination of the types of AI-generated content that she now claims she is prohibited from sharing," Judge Provinzino said. "However, Rep. Franson did not move for a preliminary injunction until Oct. 11, 2024 — more than 16 months later."
"If Rep. Franson truly believed that the statute posed a dire threat to First Amendment rights, she logically would have voted against the bill," Judge Provinzino continued. "Because the harm Rep. Franson asserts was partly of her own making, Rep. Franson's claim of irreparable harm further falls flat."
The Minnesota attorney general's office, Hancock and counsel for the plaintiffs didn't immediately respond to requests for comment late Monday.
Kohls and Franson are represented by Douglas P. Seaton, James V.F. Dickey and Alexandra K. Howell of Upper Midwest Law Center and M. Frank Bednarz of Hamilton Lincoln Law Institute.
Ellison is represented by Angela Behrens and Allen Cook Barr of the Minnesota Attorney General's Office.
The case is Christopher Kohls et al. v. Keith Ellison et al., case number 0:24-cv-03754, in the U.S. District Court for the District of Minnesota.
--Editing by Michael Watanabe.
For a reprint of this article, please contact reprints@law360.com.
