What Is Context Engineering? The Big Picture

What Is Context Engineering? The Big Picture

Every conversation you have with a large language model begins from zero. The model does not remember your name, your job, your last question, the document you uploaded five minutes ago, or the rule you asked it to follow yesterday. When you send a message, the model briefly wakes up, reads whatever text is in front of it, and then forgets everything again the moment it finishes responding. This radical amnesia is not a bug. It is the defining architectural feature of every modern LLM, and it is the reason a whole engineering discipline had to be invented to deal with it.

That discipline is context engineering. It is the practice of deliberately deciding what information enters a model's working memory at the moment of inference, and how that information is organized, prioritized, and presented. It is not the same as writing clever prompts. It is not the same as fine-tuning the model. It is not the same as building a retrieval pipeline or wiring up a tool. It is all of those things considered together as a single engineering problem, with the same goal: making sure the model has exactly what it needs, in the right shape, at the right moment, to do what you actually want.

This chapter is your starting point. By the end of it you will have a clear answer to three questions. What is context engineering, exactly? Why has it become a discipline of its own, separate from prompt engineering and model fine-tuning? And what does it look like in practice, on a real problem, with a real model? You will also leave with two reusable artifacts you can carry into every project from now on: a four-layer mental model that describes the full shape of any LLM application, and a diagnostic checklist that helps you figure out why a model is misbehaving before you blame the model itself.

Let us start with the problem that makes this discipline necessary in the first place.

The Stateless Machine in Front of You

Modern LLMs are extraordinary. They can summarize a hundred-page legal brief, generate working code in a dozen languages, translate between tongues they were barely trained on, and reason through problems that would have required a team of specialists a decade ago. But underneath all of that capability sits a brutally simple architecture: a transformer that takes a sequence of tokens as input and produces the next token. That is the whole thing. Everything else, including the apparent magic of conversation, is built on top by feeding the right tokens in and reading the right tokens out.

This architecture has three consequences that shape everything we will discuss in this book.

First, the model is stateless. It has no memory between calls. If you call the API twice with the same prompt, you will get two responses that may be similar in style but are not connected in any way the model "knows" about. The only way the model knows what your second question is about is if you literally include the previous question, the previous answer, and any relevant background inside the prompt you send. There is no internal database the model consults. There is no hidden long-term storage. If it is not in the text you submit, the model cannot see it.

Second, the model is easily distracted. The same prompt can produce wildly different outputs depending on small changes in surrounding text. Insert one irrelevant sentence at the top and the answer changes. Move a key instruction from the beginning to the middle and compliance drops. Add a few examples that contradict your rule and the model starts following the examples instead. This is sometimes called the "fragility" of LLMs, but fragility is too soft a word. The reality is that the model is doing a kind of soft pattern matching across the entire prompt, and anything that nudges the pattern in the wrong direction can derail the response.

Third, the model has finite attention. There is a hard upper bound on how many tokens it can read in a single call, and even within that bound, attention quality degrades. The famous "lost in the middle" finding, published by Liu and colleagues in 2023, showed that models are best at recalling information at the very beginning and very end of a long prompt, and noticeably worse at recalling information buried in the middle. If you stuff a hundred documents into a prompt and ask a question whose answer is in document fifty-three, the model will often miss it, even though the document is technically in the input. Length and quality of attention are not the same thing.

These three properties, statelessness, distractibility, and finite attention, are why prompt engineering alone is not enough. Prompts are short messages you write once. Real applications need to send rich, dynamic, structured information to the model on every call, tailored to the specific user, the specific question, the specific moment. That is a different kind of problem.

A small example of statelessness

Imagine you build a tiny chatbot that helps users plan vacations. On Monday, a user tells the bot, "I have a budget of two thousand dollars and I love hiking." The bot cheerfully suggests the Alps. On Tuesday, the same user opens the app again and asks, "What about somewhere warm?" The bot, having no memory of Monday's conversation, has no idea the user has a two-thousand-dollar budget or that they love hiking. It might suggest Antarctica. The user, reasonably, is annoyed.

The fix is obvious once you see it: the application must remember the conversation on Monday and include the relevant parts in the prompt on Tuesday. That is context engineering. It is not a tweak to the prompt. It is a deliberate decision about what state to keep, how long to keep it, how to summarize it, and how to inject it back into the model on every turn. The prompt is just the wrapper. The context is the system.

Defining Context Engineering, Precisely

Let us settle on a definition you can carry with you through the rest of the book.

Context engineering is the practice of selecting, structuring, and managing the information an LLM consumes at inference time, with the goal of maximizing the probability that the model produces a useful, correct, and predictable response.

Three words in that definition deserve attention, because they separate context engineering from everything else in the LLM stack.

The word selecting is about the choice of what to send. Out of all the information that exists in your system, in your database, in your user's history, in the public web, what actually goes into the prompt for this specific call? Selection is the dominant cost driver in most LLM applications. Tokens cost money. Latency rises with token count. Distractibility rises with irrelevant content. A large part of context engineering is the discipline of throwing things away.

The word structuring is about how the chosen information is laid out. Where does the system instruction go? Where do the retrieved documents go? Where does the conversation history go? In what order? With what delimiters? With what metadata? Structure is what turns a pile of text into a model-readable environment. A well-structured context allows the model to find what it needs quickly and ignore what it does not. A poorly structured context forces the model to guess.

The word managing is about the lifecycle. Context is not static. It changes every turn. Documents get added and removed. The conversation grows. Tool results arrive. The user's goal evolves. A context engineer designs the rules for how context evolves over time: when to summarize, when to forget, when to retrieve, when to ask the user a clarifying question instead of guessing.

Put together, these three words cover an enormous surface area. They cover the prompt template you write on day one. They cover the retrieval pipeline that fetches relevant documents on every call. They cover the conversation memory that persists between turns.