The RAG Reality Check: Why Your Demo Doesn't Survive Contact With Users

Every RAG system has a corpse. Somewhere in your company's Slack, in a Notion page titled "AI Strategy Q3," or in the back of a Jira epic, there is a chatbot that aced its demo. It answered questions about the company handbook. It summarized quarterly earnings. It cited its sources with confident little footnote numbers. It wowed the VP. It shipped to twenty users on a Tuesday and was quietly unplugged by Friday. The retrieval-augmented generation demo, as currently practiced, is a kind of confidence trick: it works because the inputs are chosen by the people who built it, evaluated by the people who built it, and tolerated by an audience that is, at that exact moment, hoping to be fooled.

This book is about the next part. The part where real users show up.

The Demo Lie: How RAG Projects Actually Die

There is a predictable arc to almost every RAG project that doesn't make it to production, and it rarely has anything to do with the underlying language model. The arc looks something like this.

Stage 1: The Heroic Demo. A small team — often two engineers, a product manager, and an overworked designer — spends two to four weeks building a retrieval-augmented system. They hand-pick a corpus. They hand-pick an embedding model. They hand-pick twenty queries, run them through the pipeline, polish the answers, and record a thirty-second Loom. The Loom goes viral inside the company. Someone with a VP title calls it "magical." Slack is full of fire emojis. A budget is approved.

Stage 2: The Pilot Launch. The system is bolted onto an internal wiki, a customer support inbox, or a vendor contract repository. Five to fifty people are invited to try it. The first wave of feedback is positive, partly because early users are polite, partly because they're using the system the way the demo did, and partly because nobody wants to be the person who killed the magical thing.

Stage 3: The First Real User. This is the user who asks a question the team never thought of. They paste in a paragraph from a 2019 PDF that wasn't OCR'd. They ask a multi-hop question that requires joining three documents. They ask in a language the embedding model wasn't really trained on. They ask something the corpus doesn't actually contain. The system hallucinates. The system invents a citation. The system confidently tells them the answer is in `policy_2023_final_v2.pdf` on page 14, but page 14 is a blank page.

Stage 4: The Prompt Engineering Spiral. The team's response is almost always the same: add more to the prompt. More instructions. More rules. A list of things not to do. A reminder to "only use the provided context." An apology clause. A few-shot example. A second few-shot example. A paragraph about how the assistant should respond. The prompt grows from 200 tokens to 2,000. The system gets slower, more expensive, and — crucially — only marginally more correct on the queries the team originally cared about, while remaining completely broken on the queries real users actually ask.

Stage 5: The Quiet Decommissioning. A meeting is held. The product manager explains that the AI initiative "didn't land" with users. A postmortem is written blaming the embedding model, or the chunking strategy, or "the models aren't ready yet." A new vendor is evaluated. The cycle restarts.

If you have ever shipped a RAG system, you have lived at least four of these five stages. The discomforting question is: which part of this arc is *inevitable*, and which part is a result of decisions we could have made differently? Almost all of it, as it turns out, is the latter.

The root cause is a mismatch between the loss function the team is implicitly optimizing during the demo and the loss function the system will face in production. The demo loss function is roughly: *do these twenty cherry-picked queries produce answers that look impressive in a thirty-second recording?* The production loss function is something like: *for the actual distribution of queries that real users send, in the time they're willing to wait, with the corpus as it actually exists, does the system produce answers that are faithful, useful, and defensible?* These are not the same function. They share almost no terms.

Three forces push teams toward the wrong loss function. The first is selection bias in the demo set: people build demos with the queries they know the system can answer, so the system is, by construction, good at those queries. The second is observability theater: the metrics that get instrumented during the demo phase (top-k recall on a hand-labeled set, BLEU scores against a held-out golden set) are not the metrics that predict user trust six weeks later. The third, and most insidious, is prompt-as-bandage: when the system fails, the cheapest local fix is to add text to the prompt, and so the team iterates on the wrong axis. They optimize the system's *self-description* when they should be optimizing its *behavior*.

The rest of this book is structured around the assumption that you can do better. But you can't do better without first having a clean taxonomy of the ways RAG systems fail, a working definition of what "good" means, and a checklist of decisions that have to be made before any code is written. We tackle all three in the rest of this chapter.

The Five Failure Modes

Most RAG failures, in my experience, fall into one of five categories. They are not equally common, they don't always happen in isolation, and the line between them is sometimes blurry, but they are distinct enough that you can talk about them — and, more importantly, you can design for them — separately. The five are: retrieval misses, ranking errors, context overflow, generation drift, and citation lies. Together, they form a cascade that runs from the user's query all the way through to the text the model produces.

```mermaid flowchart TD Q[User Query] --> R[Retrieval Miss] R --> K[Ranking Error] K --> C[Context Overflow] C --> G[Generation Drift] G --> L[Citation Lie] L --> A[User Sees a Confident, Wrong Answer] ```

Let's walk through each one.

Retrieval Misses

A retrieval miss is the most basic failure: the chunk the user needs is in your corpus, but you didn't pull it out. Everything downstream is wasted effort, because the model is now being asked to answer a question using context that doesn't contain the answer. The most charitable interpretation of the output will be a refusal. The most common interpretation will be a hallucination.

Retrieval misses have three main causes. The first is lexical gap: the user asked a question in their own words, and the relevant document uses different words. "How do I reset my password?" against a document titled "Account Recovery Procedure" is a classic example. Embedding models handle this better than BM25, but they don't handle it perfectly, and they are surprisingly brittle on long, formal documents where the relevant sentence is buried under hundreds of words of policy boilerplate.

The second is semantic conflation: the embedding model thinks the query and the document are similar when they aren't, or vice versa. This is especially common for queries that are short, ambiguous, or contain domain-specific jargon that the embedding model has never seen. "PTO" in a healthcare company means "paid time off"; in a manufacturing company it might mean "power take-off." A generic embedding model will happily retrieve documents about one when the user meant the other.

The third is corpus coverage: the question simply isn't in the data. This is not a retrieval failure in the strict sense, but it manifests as one to the user, and you have to decide what the system does in response. Refusing to answer is the safe default. Inventing an answer is the dangerous default. Most production systems do neither consistently.

The single biggest predictor of retrieval miss rate, in my experience, is not the embedding