The Day the Model Lied

The Day the Model Lied

On Tuesday at 10:47 in the morning, in a glass-walled conference room on the seventeenth floor of a building that smelled faintly of burnt coffee and dry-erase markers, an artificial intelligence confidently quoted a clause from a corporate travel policy that did not exist. Maya Okonkwo knew this because she had read the policy the night before — all forty-seven pages of it — and she was watching, in what felt like slow motion, as her company's brand-new chatbot fabricated Section 14.3(c) with a specificity that could have fooled an auditor. The vice president of Human Resources, a man named Gerald who was paid to be skeptical, was nodding along. The chief financial officer was taking notes. And Maya, the senior engineer who had been given ninety days to "make us look like we are ahead of this AI thing," was trying to remember whether there was a graceful way to leave the building through the seventeenth-floor window.

Then, mercifully, Gerald asked a follow-up question: "Can you show me the exact language of 14.3(c) verbatim?" The model obliged. It produced a paragraph. The paragraph did not appear in any document Maya had ever uploaded. It was not in the policy. It was not in the company's handbook, the supplemental memos, the quarterly travel updates, or the eighteen thousand emails in the legal team's archive. The model had simply invented it, and it had done so with the calm, polished confidence of a senior partner reciting case law.

The room was quiet for about four seconds. Then Gerald laughed, the CFO laughed, and Maya did not laugh because she realized — somewhere between the second sentence and the third — that her entire project was now officially a problem.

This chapter is about that problem. It is about why large language models, which can draft contracts, write poems, and explain quantum entanglement, will also lie to you with absolute conviction. It is about why the obvious solutions — "just feed it your documents," "just connect it to a database," "just turn on retrieval" — turn out to be much harder than they sound. And it is about a pattern called Retrieval-Augmented Generation, which is currently the most reliable way to make these systems tell the truth more often than not. We are going to unpack that pattern from the ground up, slowly and carefully, so that by the end of this book you can do what Maya eventually learned to do: take a powerful but ungrounded model and turn it into something that, most of the time, you would actually trust in a glass-walled conference room with Gerald in it.

The Day the Model Lied: Anatomy of a Demo Gone Wrong

Let us rewind the scene a little, because the disaster did not begin with the question about Section 14.3(c). It began six weeks earlier, when Maya's director, Priya, walked over to her desk with a printout of a McKinsey report and the kind of expression that usually precedes a budget meeting.

"Maya," Priya said, in the tone of someone who has already decided, "executives are scared of ChatGPT. They want one of those. Can we build one?"

"Can we build one" turned out to mean: build a chatbot that knew everything about the company's internal policies, could answer employee questions in natural language, did not leak confidential information to the public internet, and would be ready to demo in ninety days with a budget roughly equal to the cost of renovating the second-floor bathroom. Maya, being a reasonable engineer, said yes, because saying no was not a career option.

She did what most teams do in 2023 when asked to "build one of those." She took a strong general-purpose language model, the kind that powers the public chat tools, and built a thin web wrapper around it. She gave it a system prompt that said, "You are Acme Assistant. You answer questions about Acme Corporation's internal policies. Be helpful and accurate." Then she connected it to a small sample of documents — the travel policy, the expense policy, the HR handbook, a few FAQs — by, in essence, copying their text into the system prompt.

The first demo, two weeks in, was underwhelming. The model answered "I do not have access to that document, but you can find more information on the company intranet" to almost every question. It had not actually been given the documents in any meaningful sense; the system prompt could only hold a few thousand words, and the actual policies totaled nearly half a million.

So Maya did what the internet told her to do: she embedded the documents into a vector database, retrieved the most relevant chunks, and stuffed them into the prompt alongside the user's question. This is the canonical "RAG" pattern in its simplest form, and we will spend the next several chapters learning how to do it well. For the moment, what matters is that the demo on Tuesday used this retrieval layer. The model had access to all forty-seven pages of the travel policy. It could, in principle, find the right answer.

What happened instead was Section 14.3(c).

The question Gerald had asked was innocent enough. An employee had submitted an expense report for a one-night hotel stay in Geneva during a conference, and wanted to know whether the per-diem rules allowed the actual hotel cost, which was higher than the per-diem, because the conference hotel was the only place within walking distance of the venue. Gerald's question, paraphrased, was: "What does our policy say about hotel costs that exceed the per-diem when the event is in a city where the per-diem is unrealistic?"

The model, presented with the company's actual travel policy and the relevant expenses, returned a confident, well-structured, citation-bearing answer. It cited Section 14.3(c). Section 14.3(c) does not exist. The actual policy has no numbered subsections in Section 14 at all; Section 14 is two paragraphs long and addresses something completely different. The closest relevant clause is buried in Section 9, which discusses exceptions for "events hosted at venues where the standard per-diem is documented to be insufficient." But the model did not cite Section 9. It cited a section it had invented.

Maya understood, in the moment of Gerald's amused silence, that the word "accurate" in her system prompt was doing absolutely nothing. The model was not trying to be accurate. The model was trying to produce a plausible next paragraph, and a paragraph that confidently references a specific numbered subsection is, on average, more plausible to a reader than a paragraph that hedges with "I think the exception is somewhere in the middle of the document, possibly Section 9." Plausibility is not truth. The model was optimizing for the wrong function.

This is the central problem this book exists to solve. We are going to call it the grounding problem, and we are going to take it apart section by section until you can recognize every piece of it, predict where it will break, and design around it.

A few things deserve to be noted about the scene before we move on, because they will recur throughout the book. First, the model did not hallucinate randomly. It hallucinated in a way that was maximally convincing to a non-expert. It produced a citation. It produced specific language. It produced a paragraph that read like the surrounding paragraphs in the real policy, because the retrieval layer had handed it a few real paragraphs about hotel costs, and it had borrowed their tone, their structure, and their punctuation to fabricate a cousin. This is one of the most dangerous properties of language model hallucinations: they are not obvious nonsense. They are fluent, structured, and stylistically appropriate nonsense.

Second, no one in the room — not Gerald, not the CFO, not Maya herself — would have caught the fabrication if Gerald had not asked a follow-up. The model had done exactly what it was asked. The user got an answer that looked right. If the employee in Geneva had received that answer and relied on it, the company might have