Why Your LLM Lies: The Case for Retrieval-Augmented Generation

Why Your LLM Lies: The Case for Retrieval-Augmented Generation

In June 2023, a federal judge in Manhattan sanctioned a lawyer for filing a brief that cited six non-existent cases invented by ChatGPT. The lawyer swore under oath that he had no idea the cases were fabricated. They had the right case names, plausible holdings, and even realistic-looking citations to real reporters. The LLM had produced nothing but a confident, structured lie. The lawyer's mistake was not using an LLM; it was treating an LLM as if it were a database. The mistake is repeated, in slightly different forms, in nearly every LLM deployment that ends up in the news.

This is the failure mode that haunts every production system built on top of large language models. The model produces fluent, structured, and confident text that may or may not be true. The user has no signal to distinguish the two. The model itself has no signal either. The next token is just the most probable next token given the previous tokens, and "probable given text like this" is not the same thing as "true."

Retrieval-Augmented Generation, or RAG, is the most widely deployed architectural answer to this problem. The pitch is simple: instead of relying on what the model "knows" from training, retrieve relevant evidence from a trusted source at query time, paste that evidence into the prompt, and let the model condition its answer on it. The model becomes a reader rather than a memorizer, and reading is auditable. The user can be shown the source. The auditor can check the source. The developer can change the source.

But RAG is not a universal fix, and treating it as one is the second most common mistake people make. The first is treating the LLM as a database. The third, which we will get to, is believing that a 50-line RAG script is a production system. The fourth, which most teams discover only after launch, is that the failure modes of RAG are different from the failure modes of vanilla LLMs, and the evaluation infrastructure you need for RAG is different from the evaluation infrastructure you need for generation. A system that was working in the demo can be silently wrong in production for months before anyone notices.

This chapter is the diagnostic stage. We will go through the specific failure modes that RAG is designed to address, the failure modes it does not address, the failure modes it actively makes worse, and what a naive baseline actually does when you wire it up against a real task. By the end, you should have a clear mental model of when RAG is the right tool, when fine-tuning is the right tool, when pure prompting is the right tool, and when nothing short of changing the question will do.

1.1 The Taxonomy of LLM Failure Modes

LLMs fail in ways that look similar to the user and very different to the engineer. To a user, every error looks like "the model was wrong." To an engineer, there are at least four distinct failure modes, each with its own root cause, its own detection method, and its own preferred fix. Conflating them is why so many "we tried RAG and it didn't work" stories exist, and why so much of the discourse about RAG in 2024 was people talking past each other.

1.1.1 Hallucination: The Model Says Things That Are Not True

Hallucination is the term of art for the model generating text that is fluent, syntactically correct, semantically plausible, and factually false. The Mata v. Avianca brief is the canonical case, but hallucination shows up at every scale. A medical LLM will invent drug interactions that do not exist. A legal LLM will cite a section of the U.S. Code that was repealed in 1996. A code assistant will confidently import a library that was never published to PyPI, and the user will not find out until the import fails at runtime. A financial summarizer will invent a quarterly earnings number that sounds right but is off by a factor of ten.

The mechanism is well understood at this point. The model is a next-token predictor. At each step, it samples (or argmaxes, depending on the decoding strategy) from a distribution over the vocabulary given the previous tokens. That distribution encodes what *would* typically come next in text *like this*, not what is *true*. "Typical" and "true" overlap heavily when the model has been trained on accurate text and the question is well-formed and common. They diverge the moment the question is unusual, the topic is niche, or the training data itself was wrong. Worse, the divergence is invisible to the model. It produces a probability distribution over tokens, not a confidence over facts. A token that is probable given the model's prior is not necessarily a token that corresponds to a true claim.

There are two sub-varieties worth distinguishing because they have different fixes.

Closed-book hallucination is when the model answers from parametric memory and gets it wrong. The Mata case, before the lawyer pasted the fake cases into a brief, was a closed-book hallucination: the model was asked a question, generated an answer from what it "knew," and the "knowledge" was wrong or absent.

Open-book hallucination is when the model is given the right context and still produces a wrong answer, because the prompt did not constrain generation tightly enough, or the model lost track of the context in a long prompt, or the model "leaked" prior beliefs through the seams of the conditioning. A model that is told "the company was founded in 1842" in the context, but that has been trained on text saying the company was founded in 1942, will sometimes produce 1942 anyway, especially if the prompt is long and the conditioning information is in the middle.

RAG addresses the first variety directly and has only indirect effects on the second. RAG reduces closed-book hallucination by making the relevant facts present in the prompt. RAG does not eliminate open-book hallucination; it can even introduce new variants, because the model now has to choose between the retrieved context and its prior, and that choice is not always the right one.

A useful diagnostic: ask a model a question for which the answer changes over time ("Who is the CEO of company X?") or a question with a numerical answer that can be verified ("What is the GDP of country Y in 2022?"). If the model gets a stable, verifiable fact wrong, that is closed-book hallucination. If you paste the correct fact into the context and the model still gets it wrong, that is open-book hallucination, and a different problem. The fixes are different. Closed-book hallucination is fixed by giving the model the right context. Open-book hallucination is fixed by better prompting, better instruction tuning, better attention management, or sometimes by detecting the disagreement and refusing to answer.

1.1.2 Staleness: The World Moved On, the Model Did Not

A model trained on data through January 2025 does not know what happened in February 2025. This sounds obvious when stated that way, but the implications are routinely underestimated. Most product surfaces do not just need to be "true," they need to be "true as of right now." The model has no clock. It has no concept of "today." It can tell you the weather if you put a weather tool in front of it, but it cannot tell you the weather from memory. It can tell you who the president of the United States is, as of its training cutoff, but it cannot tell you who the president is *now*, and the difference between those two statements has, in the past, cost LLMs their credibility with users who notice.

Staleness is a particularly insidious failure mode because the model often *knows* the older fact correctly. It will tell you with high confidence that a particular senator is the senator from a state, and it will be right as of the training cutoff and wrong as of last week's election. The shape of the answer is identical to a correct answer. The only difference is a date that nobody asked about. A model cannot volunteer its staleness bec