In 2002, MCI Worldcom was charged with the largest accounting fraud in U.S. history. They capitalized regular operating expenses, making it appear as if they were allocating more toward future business investments when they were not. The financials reported to investors were analogous to AI hallucinations. The numbers weren’t real.
Current Developments With Legal Hallucinations
In 2023, the first brief to include AI-generated cases was filed. It wasn’t the first time a brief had been submitted with mistakes, but it marked a new precedent where convincing AI-generated errors could happen more often. Since that first case, there have been over 1,200 reported situations where hallucinations have made their way into filings.
In 2025, Graciela Dela Torre filed dozens of documents as a pro se litigant, allegedly using ChatGPT. Her filings were related to a recently dismissed case with her insurance company. Some documents included hallucinated cases, but the sheer volume of filings was burdensome. This March, Nippon Insurance decided to sue OpenAI regarding those filings, claiming, among other things, that OpenAI was engaging in the unauthorized practice of law.
Last week, a brief filed with the Sixth District Court of Appeals had real citations but quoted sentences that don’t appear in the cited sources. This brief was filed by an attorney who used a reputable legal research vendor’s AI offering.
AI Governance Is More Important Than Ever
The unfortunate reality is that hallucinations are a feature of LLM systems, not a bug. And they are very convincing.
Law firms need to ensure they have strong processes around the use of AI. Staff training needs to be ongoing. Greater emphasis needs to be placed on reviewing materials, as creating documents with AI is faster and easier. This needs to include who reviews, when reviews happen, and against what standard.
In short, firms need to self-monitor to ensure hallucinations of all types don’t find their way into work product. This is not limited to court filings either. Let’s not forget that contracts and legal advice provided to clients can also include hallucinations.
Legal Should Take A Cue From The Accounting Industry
The idea that critical errors can be hidden and need fact-checking is not unique to legal. Investors have long relied on a company’s financials and bookkeeping, and the accounting industry has been built around the need for trust in the numbers.
Not only are there Generally Accepted Accounting Principles (GAAP) that guide accounting, but there are also protocols for auditors to follow when independently attesting to the integrity of the numbers. The issues with MCI WorldCom and Enron resulted in the passage of Sarbanes-Oxley. The accounting industry, which had previously been self-regulated, became regulated as a result.
Now, auditors review the accuracy of the numbers and the processes used to produce them. If the processes are shaky, an auditor may be required to call out weaknesses in the controls the company has put in place.
Large businesses also have internal auditors who serve as checks and balances, identifying issues before they get to an independent auditor.
Trust
Trust is at the foundation of our financial markets and also our legal system. If it’s harder to trust and validate the veracity of a legal document, then what does this mean for our justice system?
It’s my view that we are at an inflection point where law firms and attorneys must up their game in how they review their work. With agentic solutions and client pressures, the amount of AI-assisted work product created will increase tenfold, or perhaps a hundredfold.
Validating citations and using Shepards or KeyCite is table stakes. There are now independent systems on the market that can help with citation verification and hallucinations.
More firms should incorporate processes that interrogate and use adversarial approaches to root out issues and errors in work product before a court, opposing counsel, or a client does. AI solutions can be adapted to support this function.
Organizationally, perhaps firms should consider an internal audit function that is structurally independent of practice areas, similar to those in corporations.
Dare I suggest that there may be a need for systems that serve as confidential, independent validation, similar to the role of financial auditors?
Shared Problems Benefit From Collaboration
Innovation will drive the development of solutions, especially if standards emerge.
Each firm must solve for itself, but leaders across the industry can leverage associations to work together on shared problems and best practices. (For example, the SALI Alliance is an existing forum used for data standards.)
Rule 11 And Attorney Ethics
The ABA has provided initial guidance on professional standards for AI under Rule 11. Lawyers know what they are responsible for, but they must decide how to meet those standards because there is no formal operational guidance.
The AICPA provides GAAP as guidelines for accounting and financial reporting. Perhaps the ABA might eventually offer similar guidance on operationalizing Rule 11.
What guidance should exist? And when should specific guidance begin to be offered? Can it start through industry collaboration?
Here are a few ideas for consideration:
- Citation verification? Fact-checking?
- Adversarial AI review (a second model tasked with disproving the first)?
- Sampling protocols for high-volume activity (e.g., mass tort, e-discovery summaries)?
- Document-level confidence scoring?
- Confidential and independent review?
- Human sign-off tied to defined review thresholds?
I’ve written elsewhere that innovation leads and regulation follows. But if AI innovation is going to cause friction or undermine trust in a way that can impede justice, then bar associations, regulators, or the courts may need to step in earlier.
Pro Se Litigants
Maybe courts need to consider some minimum standards before a pro se litigant can file using AI? Should there be a requirement and mechanism to disclose that AI was used in creating a filing? Perhaps the federal system or a state court could offer a service to pro se litigants to use before filing? That could mitigate some of the downsides while supporting greater access to justice.
Summary
The current legal system is engineered for accuracy, given the speed at which humans create documents. AI breaks that balance, automating more drafting and generating work product at a scale that overwhelms traditional validation methods. The review process needs automation to keep up.
Accounting has faced automation and complexity while adapting to maintain trust. Similarly, legal professionals will need tools to support more automated content creation and maintain trust in the documents they produce.
Law firms need to protect their reputations and their clients, and the legal profession needs to ensure legal documents can be trusted. Just as investors need to have confidence in financial reporting, the legal industry will need greater confidence that hallucinations are manageable when AI is part of work-product creation.
The review of AI-generated work product may be the greatest systemic limitation the legal industry faces in AI adoption today.
Ken Crutchfield has over 40 years of experience in legal, tax, and other industries. Throughout his career, he has focused on growth, innovation, and business transformation. His consulting practice advises investors, legal tech startups and others. As a strategic thinker who understands markets and creating products to meet customer needs, he has worked in start-ups and large enterprises. He has served in General Management capacities in six businesses. Ken has a pulse on the trends affecting the market. Whether it was the Internet in the 1980s or Generative AI, he understands technology and how it can impact business. Crutchfield started his career as an intern with LexisNexis and has worked at Thomson Reuters, Bloomberg, Dun & Bradstreet, and Wolters Kluwer. Ken has an MBA and holds a B.S. in Electrical Engineering from The Ohio State University.
The post What The Legal Industry Can Learn About AI Hallucinations From Auditors appeared first on Above the Law.
