AI-generated fake citations appear in over 2,000 court decisions as Google touts legal AI tool

A searchable database compiled by legal scholar Damien Charlotin documents 2,022 court decisions that incorporated AI-generated material, such as fabricated citations, between April 2023 and September 5, 2026, with the majority occurring in the United States. Google’s legal AI offering, Gemini Enterprise for Legal, was launched in August and is marketed as an augmentative tool rather than a replacement, according to general counsel Halimah DeLaine Prado, who cited its role in contract review, e-discovery and litigation prep.

The database attributes ten citations to Google’s Gemini and Bard models, while OpenAI’s ChatGPT is linked to over 100 cases. Most of the flagged incidents resulted only in judicial warnings; fewer than 160 led to disciplinary action, typically fines ranging from $1,000 to $5,000. Charlotin attributes the recent plateau in hallucinations to improved tools and greater awareness among users, noting that free large language models without internet search capabilities tend to hallucinate more. Early high-profile cases, such as a New York federal judge sanctioning a law firm for submitting AI-generated fake opinions, illustrate the growing scrutiny of AI use in legal practice.