The Context Bottleneck: Why Everything Is About Memory Now
The Context Bottleneck: Why Everything Is About Memory Now
At a Fortune 500 retailer, the on-call engineer got paged at 3:07 a.m. for a reason that had nothing to do with the model itself. A customer-support agent had started refusing to answer questions it had handled cleanly an hour earlier. Same conversation. Same user. Same underlying policies. The only thing that had changed was the length of the chat history. By turn 47 the agent had stopped remembering the customer's order number. By turn 63 it was quoting refund policies that did not exist in any documentation it had been given. By turn 80 it was asking the user, politely and confidently, to repeat information it had already collected twice.
Nothing about the model had been changed between hour one and hour three. The weights were the same. The system prompt was the same. The tool definitions were the same. What changed was the *context*. As the conversation accumulated turns, the effective working memory of the agent degraded in ways that looked, to a casual observer, like hallucinations but were, in fact, something more interesting and more dangerous: the slow structural failure of an LLM operating at the edge of its usable context.
This is the book in one sentence: in modern LLM systems, context is the new runtime. The token window is no longer a buffer you stuff with text and forget about. It is the live execution environment in which reasoning happens, in which tools get invoked, in which agents maintain identity across long horizons, and in which every architectural decision either pays a tax or earns a dividend. The teams who treat context as a first-class engineering surface — budgeted, instrumented, and versioned like any other resource — are the ones shipping reliable systems in 2025 and beyond. The teams who treat it as an afterthought are the ones paging each other at 3 a.m.
What follows is the case for taking context seriously, the mental models you need to reason about it under pressure, and the four pillars that will structure everything we discuss in the rest of this book.
---
Context Is No Longer a Buffer — It Is the Runtime
For most of the history of large language models, the word *context* meant "the prompt." You assembled a string, sent it to the model, parsed the response, and forgot about it. The model itself was the interesting artifact; the context was just a delivery mechanism. The window grew from 2,048 tokens to 4,096 to 8,000 to 32,000 to 128,000 to 200,000 to a million. Each leap was celebrated as a capability unlock, and for good reason — but the leap was almost always framed as *the model can now read more*. What it actually unlocked was far more consequential: *the model can now operate over longer-running state*.
Once an LLM can hold a million tokens in its window, you can do things with it that simply were not possible before. You can drop an entire codebase into the prompt. You can hand it a 400-page contract and ask for a redline. You can build an agent that takes 200 steps to solve a research problem and keeps its own scratchpad visible the whole time. You can give a customer-support system the entire order history of every customer it has ever talked to. The window becomes the substrate for stateful computation, not just for input.
This shift has a name in production engineering, even if we do not always say it out loud: the context is the runtime. The same way a JVM carries the bytecode of your program plus its heap plus its stack plus its thread state, an LLM call carries the system prompt plus tool definitions plus retrieved documents plus conversation history plus intermediate reasoning plus the slots reserved for the model's own output. Everything that determines the behavior of the system at step $t$ lives inside that container. If you are shipping LLM-powered software in production, the question is not "how big is the window" but "how is the runtime being managed."
This is a subtle but important inversion. For traditional software, the runtime is something you control tightly. You pick a language, a virtual machine, an interpreter, a sandbox. You know exactly how much memory you have, how it is laid out, and what happens when you exceed it. For LLM software, the runtime is something you are *handing to* a system that consumes it in a way you cannot fully observe. You can count tokens going in. You can count tokens coming out. You can even, with some effort, measure where in the context the model seemed to "look." But you cannot fully control how that context is interpreted, prioritized, or remembered. You are renting someone else's runtime and writing your program's state into a medium you do not own.
That inversion has consequences, and they compound quickly. The first consequence is that every architectural decision in an LLM system is now a context decision. Which model you pick is a context decision — different models have different effective context lengths, different attention patterns, different cost curves. How you chunk documents for retrieval is a context decision — chunk size, overlap, and embedding strategy all change what ends up in the window. How you structure tool definitions is a context decision — verbose JSON schemas consume budget that could otherwise be spent on retrieved evidence. How you summarize conversation history is a context decision — lossy or faithful, when to summarize, what to keep verbatim. How you write your system prompt is a context decision — every instruction you add has to compete for attention with every example, every retrieved passage, every prior turn. Even your logging strategy is a context decision, because verbose reasoning traces eat budget and tight traces lose debuggability.
The second consequence is that the prompt engineer has become a kind of memory architect. In the early days of LLM work, prompt engineering meant writing clever phrasing. Today, it means designing a runtime: deciding what goes into the context, in what order, with what priority, with what refresh policy, with what eviction rules, with what audit trail. The vocabulary of systems engineering — budgets, schedulers, caches, eviction policies, working sets — has become the vocabulary of prompt engineering. The teams who understand this are building systems that scale. The teams who do not are building prototypes that melt in production.
The third consequence is that naive approaches are quietly breaking in production. Every week, I see a new postmortem that follows the same shape. A team built a system that worked beautifully in a demo with a five-turn conversation. They shipped it. Real users arrived and started having 40-turn conversations, or 80-turn conversations, or conversations that pulled in 60 documents from a knowledge base, or conversations where the agent took 30 tool calls to solve a single problem. The system started to fail in ways the team had not tested for. Latency climbed. Costs exploded. Hallucinations appeared. The model started ignoring instructions it had followed reliably two weeks ago. The team blamed the model, swapped it for a bigger one, and watched the same problems appear at a slightly higher threshold.
The diagnosis in almost every one of these postmortems is the same: the context was never engineered as a runtime. It was just allowed to accumulate. The team was not managing context; they were merely filling it.
That brings us to the central thesis of this chapter, and of this book. Modern LLM systems fail in production not because the models are too small or too weak, but because the contexts they are given are too long, too noisy, too unstructured, too expensive, and too poorly instrumented. The bottleneck has moved from the model to the memory. Everything we are going to discuss in the next fourteen chapters — every technique, every pattern, every anti-pattern — comes back to this single shift.
---