Welcome to the AI Zoo: What Governance Actually Means

Welcome to the AI Zoo: What Governance Actually Means

1.1 The $500 Refund That Wasn't a $5 Can of Stain

On a Tuesday morning in March, Maria Delgado opened her phone and saw a push notification from Hartman Hardware, a regional chain of 47 home improvement stores spread across the upper Midwest. The previous afternoon, she'd ordered a gallon of dark walnut wood stain for a refinishing project on her dining room table. The can had arrived dented. Not crushed, not leaking, not dented enough to prevent her from pouring the stain out and using it, just dented. The kind of thing you might shrug at, take a photo of, send in a quick message about, and accept a five-dollar partial refund for. That's exactly what she was planning to do.

She tapped the "Help" button in the Hartman app, was connected to "Hazel," the company's new AI customer-service assistant, and typed, "My stain arrived dented. Can I get a partial refund?" Within nine seconds, Hazel responded: "I am so sorry, Maria. That's not the experience we want for you. I'm authorizing a $500 Hartman gift card to make this right. Please allow 3-5 business days for delivery to your email."

Maria stared at her phone. She read the message twice. Then a third time. She scrolled up to make sure she hadn't misread her own complaint. A $500 gift card for a $5 can of stain. Her first instinct, like that of most honest people, was to write back and say, "I think there's been a mistake." But before she could, a second Hazel message arrived: "Would you also like me to schedule a free pickup of the damaged item so you don't have to drive to a store? We can send a courier to your home on Thursday between 10 a.m. and noon."

By the time Maria was on the phone with a real human at Hartman's regional office, the gift card had been issued, the courier had been scheduled, and an entry had been logged in the company's customer relationship management system: "Reason for compensation: shipping damage. Compensation amount: $500.00. Comp type: gift card. Approved by: Hazel (AI assistant). Human approval: not required."

How did this happen? The short version is that Hartman had deployed a state-of-the-art language model as the front door of its customer service operation, had given it access to the company's order database and refund system, had trained it on several years of transcripts from its human agents, and had not given it any kind of spending limit. The slightly longer version is more interesting and more instructive, and it's the reason we're starting this book with a story about a dented can of stain.

Hartman had done almost everything right. They had picked a good model. They had fine-tuned it on real customer interactions, which is generally a smart move because it teaches the AI the tone and vocabulary that real customers use. They had given it just enough context to do its job: it could look up orders, see shipping status, check refund eligibility, and process gift cards up to a certain amount. They had built a friendly persona, "Hazel," because the marketing team had read research suggesting that customers respond better to AI agents that have a name and a small backstory than to faceless bots. They had rolled it out gradually, starting with a soft launch in two stores, monitoring transcripts, and gradually expanding.

What they had not done, because at the time they didn't know they needed to, was treat the AI as a system that needed governing. They had governance for their payment processing system, which handled thousands of dollars a second. They had governance for their inventory system, which moved millions of items through warehouses. They had governance for their employee data, governed by HR policies and legal review. But for Hazel, the AI assistant that could talk to customers and hand out money, the only "governance" was the original product requirements document and a Slack channel where engineers sometimes argued about prompts.

A few things made the $500 refund possible, and all of them are classics of what happens when AI is deployed without governance. First, the model had no awareness of proportionality. It had been trained on thousands of conversations in which human agents had offered a small gift card, a discount on a future purchase, a sincere apology, or sometimes, in cases of severe frustration, a more substantial gesture. The model had learned the *pattern* of escalating compensation when a customer seemed upset, but it had no internal meter that said, "Wait. The item cost $5. Anything above $5 is already more than the value of the item. We should slow down." That kind of common-sense arithmetic is trivial for a human, but it requires a rule to be in place for a model, and Hartman had not put that rule in place.

Second, the model had no business limits. A human agent at Hartman was allowed to issue refunds up to $50 without a supervisor's approval. Above that, they had to click a button that paged a manager, who would review the case and either approve or modify the request. Hazel had been given the technical ability to issue a refund, but no business rule capped that ability. When it generated the $500 figure, the system simply processed it.

Third, and perhaps most important, no one was watching in real time. The transcripts of Hazel's conversations were being stored, and a sampling of them was being reviewed by a human once a week, but no one was sitting in front of a dashboard watching the AI make decisions as it made them. The system had no alerts, no kill switch, no automated check that said, "This compensation is 100 times the value of the damaged item. Maybe pause and ask a human."

By the time Maria called the regional office to point out the error, Hartman had already given out eleven other $500 gift cards that week, including one to a customer who had complained that a screw had been missing from a package, and another to a customer whose delivery had been delayed by 30 minutes. The total unplanned cost: $6,000. The total reputational upside: roughly zero, because none of those customers were going to go on social media and post, "Wow, this hardware store gave me way too much money for my minor complaint!" People who receive windfalls tend to keep quiet about them. People who feel mistreated tend to tell the world.

This is the kind of story that makes AI governance suddenly feel less like an abstract concern and more like an urgent one. Hartman is not a hypothetical company, even though I've changed its name; the situation it faced is happening right now, in 2025, at companies in every industry. The bot doesn't have to be a customer-service bot. It could be an AI assistant scheduling surgeries, an AI tool approving loan applications, an AI agent writing and sending emails on behalf of executives, an AI model generating code and pushing it to production. The shape of the story is the same: a system that is more capable, more autonomous, and more expensive than its owners fully understood, doing things that no one designed it to do, in a way that no one noticed until the bills came in or the headlines broke.

The good news is that none of this is inevitable. The $500 refund could not have happened if Hartman had put in place five simple things, which we will explore in detail in this chapter: a permissions structure that said "an AI cannot issue a refund above $25 without human approval," an auditing system that flagged unusual transactions in real time, a cost-monitoring layer that showed daily spending, a compliance checklist that walked the AI through legal and ethical considerations before each action, and a lifecycle process that required periodic re-evaluation of the model. None of these are technically difficult. All of them are organizationally easy to skip, which is exactly why most companies do skip them, at least at first.

This chapter is the foundation of the book. Before we get into the technical and organizational details of how to govern AI, we nee