The governance wedge · Article 6 of 7

The tool is never the one who is responsible

Fabricated citations now draw sanctions, and the sanctions now draw discipline. Human oversight counts for something only when “reviewed” is defined, budgeted, and signed.

Adam Simon · Senior Technical Consultant · August 2026

1 · The cases have names now

On September 12, 2025, a California appeals court fined a lawyer $10,000 for briefs built on cases that do not exist. The court published the opinion deliberately, which is how a routine sanction becomes a standing warning to everyone else. Presiding Justice Lee Edmon wrote that “nearly all of the legal quotations in plaintiff’s opening brief, and many of the quotations in plaintiff’s reply brief, are fabricated,” and that the fake authority went undetected “because he did not read the cases the AI tools cited.” The clerk was ordered to notify the State Bar. The court explained why it published: “we publish this opinion as a warning,” and set the standard in one sentence, that no paper filed in any court should contain citations “that the attorney responsible for submitting the pleading has not personally read and verified.”

How the briefs were built is the part worth sitting with. Counsel drafted them himself, “enhanced” them with ChatGPT, then ran the result through other AI platforms to check for errors. He did not read the enhanced version before filing it. Every step of that workflow had a review in it. None of the reviews was a person.

The pattern is older than that opinion. In June 2023 a New York federal judge sanctioned two lawyers over a brief resting on decisions that had never been handed down, and framed the duty in one line that has aged well: “existing rules impose a gatekeeping role on attorneys to ensure the accuracy of their filings.” What is new is the tail. In February 2026 a State Bar Court judge recommended that the lawyer in the California case be suspended for a year, with the suspension stayed and a year of probation, conditioned on ten hours of continuing education on technology, at least half of it on the risks of using AI. The professional consequence outlived the fine.

The governing ethics guidance says the same thing without waiting for anyone to be caught. The American Bar Association’s Formal Opinion 512, issued July 29, 2024, is advisory rather than binding until a state adopts it, and it is unambiguous on where responsibility lands: “regardless of the level of review the lawyer selects, the lawyer is fully responsible for the work on behalf of the client.” It goes further than the individual. Managerial lawyers, it says, “must establish clear policies regarding the law firm’s permissible use of GAI.” Not good intentions. A policy, written down, that somebody owns.

Set aside that these examples are legal ones. Courts simply happen to be the one workplace that publishes its errors, and the published numbers are not small: of the 23 case quotations in that opening brief, 21 were fabricated. The mechanism is universal, and every reputation-first firm runs along the same fault line. A publisher’s reference entry with a confident invented date. An accountant’s memo citing a provision that reads slightly, materially wrong. A consultant’s market figure that no source contains. Opinion 512’s phrasing for the failure mode is exact and worth keeping: these tools are “prone to ‘hallucinations,’ providing ostensibly plausible responses that have no basis in fact or reality.”

Convincing is the operative word, and it is the whole difficulty. Bad work that looks bad gets caught by the ordinary machinery of a professional firm, which is built to notice sloppiness. This looks good.

2 · Why the machine invents

A language model is a fluency engine, not a filing cabinet.

You do not need the mathematics. One mechanical fact explains every incident above: a large language model generates the most plausible next words given everything so far. Plausibility is the objective. Plausibility and truth travel together most of the time, which is what makes these tools remarkable. They part company without warning, which is what makes them dangerous, because the model’s confidence does not drop when it crosses the line. The register never changes. An invented case citation arrives in the same assured tone as a real one, correctly formatted, plausibly named, entirely hollow.

That also tells you where the danger concentrates. The highest-risk outputs are the most specific ones. Citations. Quotations. Numbers, names, dates, provisions. Which is to say the exact material a professional deliverable is made of, and the exact material a reader cannot validate by feel.

Newer techniques that ground models in retrieved documents shrink the problem, sometimes dramatically, and every serious vendor is working on it. Nothing published to date removes it. Which means the control cannot sit inside the tool at all: it has to sit in your process, where a person can be held to it.

3 · Define “reviewed,” because a glance is not one

Every firm using AI claims human oversight. Ask a partner what reviewed actually means in their firm, though, and what you generally get is a pause, followed by a description of something closer to proofreading.

There was a human in the loop in each of the incidents above. The loop consisted of reading the output and finding it convincing, which, per the mechanism, it always is. Review that only asks whether the output reads well is grooming rather than oversight.

A review standard worth the name is written down, and it is keyed to types of claim rather than to how the draft felt on the read. Ours changes a little from engagement to engagement and still fits on one page.

• Citations and references: pull the source. Every case, statute, article, and ISBN gets opened rather than pattern-matched. If the source cannot be produced, the claim comes out.
• Quotations: read them where they were first published, not where the draft found them. Models paraphrase fluently and attribute loosely, and a quotation that has been smoothed is worse than one that is simply wrong, because it survives a careful reading.
• Numbers: recompute them, or trace each one back to a source document somebody can open. Plausible arithmetic is the model’s specialty.
• Names, dates, and provisions: verify against the record. Small mutations, an eighth edition that is really the seventh, are the commonest species by far.
• Reasoning and judgment: a reviewer qualified in the subject reads for what is missing rather than only for what is wrong. The costliest failures are omissions, and no citation check catches those.

Two properties let a standard like this survive contact with a busy week.

It is proportionate: tier-one internal work gets a lighter pass than a client deliverable, matching the risk tiers this series set up earlier. Reviewing everything to the same depth is how a standard gets quietly abandoned in March.

And it is budgeted. Verification time is planned into the task rather than stolen from it. If AI drafting saves three hours and diligent review costs one, the win is real and it is honest. Pretending the review costs nothing is how firms slide back to the glance.

4 · Gates, not vibes

Process beats intention, so put the standard into the workflow as gates: fixed points where work cannot proceed until a named person has done a defined thing.

The first gate is verification against the standard above, by somebody qualified to catch what is wrong in this particular subject matter. The second is ownership. A named person signs the deliverable out, and the record shows who. Never a committee, and never the software. Our readiness checklist states the principle as bluntly as we know how: accountability for AI-assisted work sits with a named person rather than the tool, because the tool cannot be embarrassed, sanctioned, or sued, and everyone else involved can.

Make the gates visible. Initials on the review, a one-line log per deliverable, an occasional sampled audit where a partner pulls a finished piece and walks its verification trail. Lightweight is fine. Invisible is not, because invisible review reverts to the glance under deadline pressure, which is precisely when the model’s worst output ships.

5 · When it fails anyway

Something will eventually pass the gates. The standard for a governed firm is not zero errors; it is errors caught fast and handled well, which takes three decisions made in advance. How an error, once spotted, gets corrected, and who tells the client. How the gate that missed it gets adjusted. Where the incident is recorded, so the pattern becomes visible over time.

A firm that can show a client the error, the fix, and the changed process has often deepened the relationship. A firm that hopes nobody noticed has bet the brand on hope.

Where this fits

Oversight is the fifth of eight domains in our AI Governance Readiness Checklist, and it is the one your clients price silently every time they read your work.

The next article takes up the question clients are starting to ask out loud, which is how you use AI, and why a firm with real gates should want to answer it rather than dodge it.

One limit, stated plainly. We design and build review workflows, and we build automation with these gates welded in. We do not practice law, and nothing here is advice about your professional obligations. The narrower promise is the useful one. If your current review standard would not survive the words “show me,” we can help you build one that would.