What Even Is AI Governance (And Why Your CFO Already Cares)

What Even Is AI Governance (And Why Your CFO Already Cares)

The CFO didn't ask about your prompt engineering. She asked about the $2.1 million line item that appeared on next month's cloud-inference bill, and nobody on the engineering org chart could explain it. Welcome to AI governance.

This book is a field guide. It is written for the engineer who got handed a "responsible AI" charter on a Friday afternoon, the architect who has to pick a model-serving platform by next quarter, the head of risk who keeps getting asked about the EU AI Act, and the chief data officer whose model catalog is, frankly, a security incident waiting to happen. If any of those are you, the good news is that the playbook exists. It is just not yet in one place. The better news is that none of it is exotic. The entire discipline can be built from off-the-shelf patterns, the same way you would build any other control plane. The hard part is the sequencing.

This first chapter is the orientation. We will look at three real failure modes, extract the principles that failed in each, build a vocabulary that the rest of the book will use, and end with a one-page exercise you can run inside your own organization before you read another word. By the end, you will not have a complete governance program. You will have a map of the territory and a starting compass bearing. The rest of the book is a guided tour through the architecture, the process, and the tooling that turn the map into a working program.

Let's start with the cases. They are short, slightly composite, and very ordinary. That is the point. None of them required a genius attacker. All of them required the same absence: nobody owned the decision.

Three Failure Modes (and What They Teach Us)

Case 1: The Memorized Patient. In late 2023, a healthcare analytics firm fine-tuned an open-source 7-billion-parameter language model on eighteen months of de-identified clinical notes. The intent was unobjectionable: build an internal assistant that could summarize patient encounters for the care-coordination team, freeing nurses from the worst of the chart-review grind. The de-identification pipeline stripped names, addresses, social security numbers, and obvious identifiers. It left the clinical content intact, because that was the point. The data scientist pushed the model weights to a shared team space on a public model hub so the rest of the analytics group could iterate without re-running the training job. They filled out the model card with a two-line description and a vague reference to the de-identification process. They did not run a memorization audit. They did not test whether the model could be induced to regurgitate training data. They did not, crucially, restrict who could download the weights.

Three weeks later, a security researcher — the kind that does this for fun on weekends — demonstrated that carefully crafted prompts could elicit near-verbatim fragments of the original notes. One query returned a detailed MRI report with the patient's initials embedded in a free-text field the de-identification script had missed. Another returned a sequence of oncology follow-up notes that, when cross-referenced with public obituary data, re-identified three individuals to a confidence that would have held up in a courtroom. The HIPAA breach clock started. The OCR notification went out. The eventual settlement was north of four million dollars, on top of the legal fees, the rebuild, and the reputational hit. The model itself was quietly taken down a week later. The team was reassigned. A new architecture was commissioned, on a different stack, with a different team, and with a different set of rules.

The lesson is not "fine-tuning is dangerous." Fine-tuning is fine. The lesson is that the model inherited the entire training corpus — including everything that was supposed to have been removed, and everything that the de-identification pipeline had not been thorough enough to remove — and nothing in the workflow asked the model to forget. The lesson is also that the weights were published to a place that was effectively public, that the model card was empty, and that nobody asked whether the data inside the model was the kind of data that was allowed to leave the building. The fine-tune itself was not the failure. The absence of memory hygiene, the absence of egress controls, and the absence of an asset inventory were the failure.

Case 2: The Hallucinated Citation. You almost certainly know this one, but it is worth retelling because the second-order failure is what matters. In 2023, an attorney representing a plaintiff in a personal-injury suit against an airline filed a brief citing six prior federal decisions supporting the motion. The opposing counsel searched. None of the cases existed. They had realistic captions, plausible docket numbers, and quotation marks around fabricated language — a perfect simulacrum of legal authority, generated by a chatbot in response to a request for "cases supporting this argument." The judge imposed sanctions on both the attorney and the firm. The headlines read "Lawyer Used ChatGPT," which is the wrong story. The right story is that the firm had no policy on the use of generative AI in legal work, no tool restriction in place for the practice groups, no peer-review checkpoint for AI-generated content, and no audit trail for which AI tool had been used to draft which document. The attorney was not being reckless. They were being optimistic. The system had no spine, and when a junior lawyer used the most accessible tool on the market to do a routine research task, the system did not catch it.

The rebuild took six months. The firm added review gates to the document lifecycle. They added an approved-tools list, with technical controls rather than just policy language. They added a model-output-classification step to the document management system, flagging content that showed statistical signatures of AI generation. They added training, which was the easy part. The hard part was admitting, in writing, that none of this had existed before the incident. The liability tail, meanwhile, will run for years. The client relationship survived, but only because the plaintiff ultimately settled. Had the case gone to trial, the firm's exposure would have been materially worse.

Case 3: The Runaway Invoice. In December 2023, a mid-sized SaaS company — call it ProjectOps, though that is not its real name — shipped an AI customer-support assistant into production. An intern, working under a deadline, configured the model with a generous token limit and a high temperature to make responses feel "more conversational." The intern did not consult anyone, because the configuration was a single screen in the deployment console and the change felt trivial. The intern left for the weekend. The auto-scaler did its job. By Monday morning, the assistant had handled roughly eleven million conversation turns. Many of them were unusually long: users discovered that the assistant would engage in extended philosophical, creative, and — it later emerged — occasionally romantic exchanges, and the assistant was willing to oblige. Several API retries compounded the cost. A pricing change at the upstream model provider, announced two weeks earlier and buried in a vendor newsletter, had shifted the unit economics in a way that nobody on the engineering team had tracked. The single-line invoice arrived the following week: $2.1 million.

The CFO called a meeting. Engineering scrambled to add rate limits, token caps, and per-user quotas. The board asked three questions, in order: *Who approved this deployment? Who is monitoring it in real time? What is our exposure to the next one?* Nobody could answer any of the three with confidence. The intern's configuration was reverted. The model was downgraded to a cheaper tier. A "cost dashboard" project appeared on the next planning cycle, sponsored by finance rather