Why Context Is the New King

Why Context Is the New King

The model was state-of-the-art. The benchmark scores gleamed. The demos looked like magic. And then you shipped it — and the moment real users touched it, the magic started leaking out of every seam.

If you have ever put a powerful language model into production and watched it fumble, hallucinate, contradict itself, or quietly forget what a user said three messages ago, you already know the feeling: something is wrong, but the model itself seems fine. The same model passes reasoning tests, scores top marks on coding benchmarks, and writes essays that would not embarrass a college freshman. So why does it suddenly start making things up? Why does it seem to have the memory of a goldfish wrapped in fog? Why does it confidently cite papers that do not exist, or recommend functions from files it never read?

The answer, almost always, lives in a place that nobody is looking: the context.

This chapter is the front door of the book. It is where we set the stakes. We will not yet talk about context windows, retrieval-augmented generation, or any of the clever machinery that later chapters will pull apart and rebuild. We will do something more fundamental first. We will convince you — with three real-ish stories, a useful framework, and one hands-on exercise — that most of the LLM failures you have ever cursed are not model failures at all. They are context failures. And once you see them that way, a whole new discipline opens up: context engineering.

The Forgetful Genius: Three Tales of Context Gone Wrong

Let us start with three small stories. They are composites, stitched together from a hundred postmortems, but they are honest. Every engineer who has shipped an LLM product has lived at least one of them, and probably all three.

Story One: The Support Bot That Lost the Order

Picture a customer-support assistant deployed by a mid-sized online retailer. The team picked a top-tier model, wrote a careful system prompt, ran it through dozens of test scenarios, and watched the offline evaluation scores come back close to perfect. Everything looked fine in the lab.

Then a real customer named Priya showed up with a real problem. She opens a chat and says:

> Hi, I want to check the status of my order #A-88421. It was supposed to arrive yesterday but it didn't, and now I need to update the delivery address before it actually ships.

The bot politely greets her, thanks her for reaching out, and asks how it can help today — as if she had not just told it. Priya, mildly confused, repeats:

> My order. A-88421. Where is it?

The bot now answers correctly, pulling a (mock) order-tracking function and reporting that the package is in transit, with an expected delivery of tomorrow. Prika is relieved, and asks:

> Okay, can I change the delivery address to my office?

The bot walks her through the address-change flow. So far, four messages into the conversation, things are still working. Priya then asks a clarifying question about whether the change will apply before the shipment, and after the bot answers that, she says:

> Actually, wait — what was the order number again? I want to write it down for my records.

And the bot replies:

> I'm sorry, I don't have access to your order number. Could you please share it with me so I can look it up?

Priya is annoyed. She literally told the bot the order number in her very first message. Worse, the answer is sitting right there in the conversation history. The model could not see it — or rather, the system feeding the model could not see it. Somewhere between message one and message five, the conversation history got truncated. The earliest turns were silently dropped to make room for new content, and the order ID went with them.

This is a textbook case of what we will later call an omission bug: the right information was once available, but it fell out of the context before the model needed it. Nothing about the model was broken. The retrieval was broken. The memory was broken. The context pipeline was broken. And the consequence was not just a wrong answer — it was a customer trust problem, because the bot came across as incompetent about something it had been told in plain English.

Story Two: The Code Assistant With the Wrong File

Now imagine a senior developer, let's call him Marcus, using an AI coding assistant hooked into his IDE. He has been hunting a subtle bug in `auth.py` for the better part of an hour. The function in question, `verify_token`, returns 401 even on tokens he knows are valid. He highlights the function and asks the assistant: "Why is this function returning 401 even when the token is valid?"

The assistant cheerfully responds by analyzing `auth_utils.py` — a different file entirely — and suggests that the bug is in how the helper validates signatures. The assistant helpfully produces a diff and an explanation. Marcus is confused. He has looked at `auth_utils.py` already; it is fine. He tries again, more pointedly: "I mean the function in auth.py. The one I have open right now."

The assistant doubles down, still talking about `auth_utils.py`, this time with even more confident reasoning about signature validation. Marcus opens the file himself and sure enough, the assistant is looking at the wrong file entirely. The wrong file's contents were injected into the model's context. Worse, because the assistant's response sounded plausible, a less experienced engineer might have wasted an hour "fixing" code that was never broken, possibly introducing new bugs in the process.

The root cause was a flaky file-resolution step in the IDE plugin. When the user said "this function," the plugin asked the editor for "the active file," but the editor's notion of "active file" and the user's mental model of "active file" did not match. The user was thinking of `auth.py` — the one he had highlighted — but the editor's cursor was last in `auth_utils.py` from a previous edit. So the IDE dutifully retrieved `auth_utils.py` and sent it to the model. The retrieval was correct relative to what was asked. The pipeline was completely wrong relative to what was meant.

This is a contamination bug: the wrong information is sitting in the context, confidently crowding out the right information. The model is doing its job — generating a coherent response based on what it sees. The pipeline is feeding it garbage, and the model cannot tell the difference between garbage and signal.

The really pernicious thing about contamination is that the failure mode is invisible from the outside. The assistant's output was fluent, plausible, confident. It even contained real-looking code suggestions. If Marcus had been less senior — if he had not already known that `auth_utils.py` was fine — he might have taken the suggestion at face value and lost an afternoon to a wild goose chase.

Story Three: The Research Agent That Invented Papers

Last one. A graduate student uses a research agent to do a literature review for her thesis chapter on memory-augmented language models. She asks: "Find me recent papers on memory-augmented language models and summarize the main approaches."

The agent fires up its retrieval pipeline, searches an academic index, and pulls in fifteen abstracts that have some keyword overlap with the query. Then it generates a beautifully structured summary organized into three categories of approach: external memory modules, episodic memory, and parametric memory scaling. The summary is well-written, confident, and cites specific paper titles. The problem: at least three of the papers it cites do not exist. The titles sound plausible — "Episodic Memory Transformers for Long-Horizon Reasoning," "Scaling Parametric Memory Beyond Context Length" — but try to look them up and you find nothing, or worse, you find completely different papers by the same authors.

What went wrong? Two things, layered on top of each other. First, the retrieval query was too broad. "Recent p