There is a certain dark comedy in watching the law firm that advises OpenAI on its “safe and ethical deployment” of artificial intelligence rush to the federal bankruptcy docket seeking leniency after realizing they’ve filed a lengthy brief riddled with AI hallucinations. It doesn’t matter where on the Am Law 100 food chain you are, the AI psychedelic experience can come for anyone who lets their editorial standards slip.
In a letter dated Saturday, Sullivan & Cromwell partner Andrew Dietderich wrote Chief Bankruptcy Judge Martin Glenn of the Southern District of New York a letter that will live forever in the Biglaw Hall of Hilarity. Dietderich informed the court that he had learned on Thursday that the firm’s emergency motion in the Chapter 15 case of Prince Global Holdings — the BVI-incorporated husk of a Cambodian forced-labor scam conglomerate — had gotten high on its own supply of AI tools.
The inaccuracies and errors in the Motion include artificial intelligence (“AI”) “hallucinations.” “Hallucinations” are instances in which artificial intelligence tools fabricate case citations, misquote authorities, or generate non-existent legal sources. We deeply regret that this has occurred. The Firm maintains comprehensive policies and training requirements governing the use of AI tools in legal work. These safeguards are designed to prevent exactly this situation. The Firm’s policies on the use of AI were not followed in connection with the preparation of the Motion. In addition, the Firm has general policies and training requirements for the proper review of legal citations. Regrettably, this review process did not identify the inaccurate citations generated by AI, nor did it identify other errors that appear to have resulted in whole or in part from manual error.
We hear a lot about the “safeguards” that lawyers put up around AI, but at the end of the day it feels like empty PR talk for “we like to think we’re editing what we send out the door.” Which, in fairness, is the ultimate safeguard. There are great tools out there designed to reduce the risk of a pernicious hallucination. And these tools will give lawyers a leg up when it comes to tamping down errors early. But there’s no substitute for a junior having to print up the cases and do the meticulous checking… and then the midlevel doing the exact same thing. That’s the manual review process the letter references. It’s inefficient, but perfection isn’t intended to be cheap and easy.
That is, in fact, why someone hires Sullivan & Cromwell in the first place.
Dietderich goes on to explain at some length — the firm has a whole program. Two required training modules. Tracked completions. Office Manual language instructing lawyers to “trust nothing and verify everything.” Policies! Mandatory training! Verification requirements!
And yet.
This is what we’re talking about when we say AI is in the process of making lawyers dumber. In the earliest days of AI, it was easy to put all the blame on the human lawyers failing to maintain best editing practices. But as the technology advances and AI providers boast that they’ve automated more and more steps in the process, the human can enter the workflow later in the game and that can subconsciously undermine the editing approach. We don’t know how automated the S&C workflow is, but it’s all a continuum — once AI joins the workflow, the clock starts ticking on someone looking at fully artificially generated work product and taking a slightly lighter red pen to the output. Before long, that’s going to miss something.
Schedule A to the letter catalogs the damage across the motion, the verified petition, the joint administration motion, the scheduling motion, and a couple of declarations — roughly 40 corrections. Some substantive, some less so, all embarrassing. The fixes include wrong pin cites, wrong volume numbers, parenthetical quotes that just… aren’t in the cases.
Build all the AI policies you want, but there is no substitute for having a human — preferably multiple humans — print everything out, take a ruler and a red pen, and go line by line cross-checking everything. It’s tedious work for the lawyers and expensive work for the clients, but it’s better than having to write… this letter.
This is not a Sullivan & Cromwell problem. This is a profession problem. We have been covering AI hallucination sanctions cases so relentlessly that Damien Charlotin has now catalogued over a thousand of them. Tools like BriefCatch’s RealityCheck exist specifically because firms can’t be trusted to run this validation themselves. The problem cuts across the law firm equivalent of class lines. It’s not just overwhelmed solo practitioners or overeager mid-tier firms looking to punch above their weight. Every firm will deal with this soon enough. They can either accept that using AI on the front end doesn’t alleviate the back end labor or they can accept writing letters to judges after the fact.
One last thing. The errors in the Chissick Declaration include Bluebook-style corrections to citations of Amnesty International and UN human rights reports about forced labor and trafficking in Cambodia. Those aren’t AI hallucinations, but the simple human mistakes that arise from embracing the unhinged rules memorialized in the Bluebook. But they are a reminder of what this case is actually about: people held behind barbed wire and forced to run pig-butchering scams. The JPLs are trying to trace billions of dollars in crypto to make victims whole. They need a recognition order to do it. And they lost a couple of weeks of runway because S&C didn’t check its work.
(Check out the full letter on the next page…)
Earlier: Has AI Managed To Make Lawyers Even Dumber?
Biglaw Firm ‘Profoundly Embarrassed’ After Submitting Court Filing Riddled With AI Hallucinations
Am Law 100 Firm Accused Of Filing Brief Riddled With AI Hallucinations… AGAIN!
Prosecutorial Error: ‘AI Hallucinations’ Are Popping Up In Criminal Cases
New Tool Catches AI Hallucinations In Legal Briefs
Understanding AI Hallucinations: Making Sure You Don’t End Up At The Wrong Stop
Legal AI Might Be Accurate… And Still Not Right
Joe Patrice is a senior editor at Above the Law and co-host of Thinking Like A Lawyer. Feel free to email any tips, questions, or comments. Follow him on Twitter or Bluesky if you’re interested in law, politics, and a healthy dose of college sports news. Joe also serves as a Managing Director at RPN Executive Search.
