Why AI Governance Stopped Being Optional

Why AI Governance Stopped Being Optional

A federal judge in New York was reading through a ten-page legal brief filed in early 2023 when something felt off. The cases cited by the plaintiff's attorney looked strangely familiar, and a quick check revealed that they did not exist. The attorney, Steven Schwartz of Levidow, Levidow & Oberman, had used ChatGPT to draft his filing. When asked about the hallucinated citations (fabricated but confidently presented as real), Schwartz told the court he was "unaware that its content could be false." The judge, P. Kevin Castel, was less forgiving: he sanctioned Schwartz and his partner Peter LoDuca, referred the matter to the relevant bar associations, and ordered them to send letters of apology to the real judges whose names had been attached to the fabricated opinions. The case, *Mata v. Avianca*, became a global punchline, but for general counsels and risk officers it became something else entirely: proof that the models their employees were already using could land them in court.

A few months later, on a Friday evening in early 2024, the head of platform engineering at a mid-sized SaaS (Software as a Service) company got a Slack message from finance that made his stomach drop. A misconfigured retry loop on a customer-facing chatbot had been hammering a third-party language model API for seventy-two hours. The bill arrived on Monday: $187,000 in inference charges for what should have been a $400 monthly line item. The bug, a one-line Boolean inversion in a denylist, was easy to fix. The conversation with the CFO, less so.

A few hundred miles away, in March 2023, the Italian data protection authority Garante had already issued something that, until then, seemed unthinkable: a temporary ban on ChatGPT in Italy, citing unlawful processing of personal data and absence of an age-verification mechanism. OpenAI took the service offline in the country within hours. Other regulators in Germany, France, Spain, and Canada opened their own investigations within weeks. By the end of 2024, the Italian order had been softened but never fully rescinded, and a new pan-European law, the EU AI Act, was already in force, formalizing the principle that artificial intelligence systems are not exempt from the rule of law.

Three incidents. Three different failure modes. One common root cause: nobody was in charge of the AI. This chapter is about why that root cause is suddenly a board-level problem, and what the rest of this book will help you do about it.

The Three Failure Archetypes (and What They Teach Us)

It is tempting to dismiss any one of those incidents as an outlier, an aggressive lawyer, a sloppy engineer, an activist regulator. They are not. They are representative of the three failure modes that, between them, account for the overwhelming majority of enterprise AI incidents in 2024 and 2025: factual hallucination, runaway cost, and regulatory non-compliance. Each is worth examining in detail, because each maps to a different pillar of the governance program you will be building by the end of this book.

Archetype 1: The Hallucinated Truth

Hallucinations, meaning confidently asserted, factually wrong outputs, are not a bug in the modern sense. They are a structural property of generative language models. A model is, in the strictest sense, a high-dimensional autocomplete. It produces the next token (a token being roughly a word fragment, the basic unit the model reads and writes) that is most probable given everything it has seen so far. There is no internal fact-checker. There is no "wait, am I sure?" subroutine. When the model produces "Vargas v. TransUnion, 2021 WL 1718230 (S.D.N.Y. 2021)" in a legal brief, it is doing exactly what it was trained to do: producing text that looks like a citation. The fact that the case does not exist is, from the model's perspective, indistinguishable from the fact that the case does.

This is not a problem that will be solved by a model upgrade. The frontier labs have made meaningful progress on hallucination rates. OpenAI's GPT-4o hallucinates less than GPT-3.5, Anthropic's Claude 3.5 Sonnet less still, Google's Gemini 2.0 family has explicit "grounding" features that check outputs against retrieved sources. But no production-grade model has a hallucination rate of zero, and the public benchmarks (TruthfulQA, SimpleQA, HaluEval) measure something narrower than what an enterprise actually cares about. What an enterprise cares about is: *for a specific question in a specific domain, will this model give an answer that is verifiable, defensible, and safe to act on?* That is a context-specific question, and no off-the-shelf model can answer it for you.

The *Mata v. Avianca* case is the canonical hallucination cautionary tale, but it is no longer the only one. In 2023, a real estate agent in California reportedly submitted a listing description generated by ChatGPT that invented facts about a school district; the seller sued. In 2024, a financial analyst at a regional bank was found to have used a generative tool to draft an investor presentation that included a fabricated earnings forecast traceable to no public filing. The bank settled with its regulator quietly, behind closed doors. None of these stories made the front page, but every compliance officer in America has heard whispers of similar incidents in their own sector.

The governance lesson is straightforward: the model is not the source of truth; your retrieval and verification layer is. The pattern that has emerged across high-functioning enterprise deployments is sometimes called RAG-and-verify (RAG stands for retrieval-augmented generation, a design pattern in which the model is fed relevant documents retrieved from a trusted store before it is asked to answer). Instead of asking the model to answer from memory, you ask it to answer from a curated corpus of approved sources, and you require it to cite those sources. When the sources do not support the claim, the model is, in effect, told not to claim it. This does not eliminate hallucinations, but it makes them auditable. And auditable is the difference between "we trust the AI" and "we trust the AI, and we can prove we should."

Archetype 2: The Exploding Bill

The SaaS company's $187,000 weekend is not unique. There is now a small but growing body of public postmortems (and a much larger body of private ones, shared in hushed tones at FinOps, or Financial Operations, conferences) of inference bills that ran into six or seven figures before anybody noticed.

The shape of the failure is remarkably consistent. A developer builds a small internal tool that calls a model API. The tool is useful, gets shared, gets embedded in a workflow, gets called by other workflows. Nobody adds rate limiting, because the tool is "internal" and the budget is "small." Then one of three things happens: a request loop in production code, a misuse case that generates millions of tokens (a model's input or output chunks of text, typically priced per million), or a billing model change by the API provider that the customer's monitoring never picked up. By the time finance notices, the month's budget is gone.

This is not a hypothetical. In 2023, OpenAI itself reportedly had to refund a small number of customers whose usage had been miscategorized by a backend change. In 2024, a Y Combinator founder publicly posted about a $50,000 bill incurred over a single weekend by a chatbot that had been incorrectly configured to handle a customer support queue. The single most cited rule of AI cost management in 2025 is one that any cloud engineer would recognize: *never put an unmonitored, uncapped, unauthenticated endpoint in front of a paid model API.* If that sounds like table stakes for cloud infrastructure, it is. The problem is that the people building the AI are not the people who built the cloud infrastructure, and the controls that the cloud team spent ten years learning to love h