The Discipline of Context Engineering

The Discipline of Context Engineering

Every token that reaches a foundation model is the residue of a decision someone made — whether to include it, where to place it, how to compress it, when to refresh it, and how to retire it. The naive mental model of "prompt = what the user typed" hides this decision layer entirely. In practice, the input a model sees in production is the output of a pipeline: ingestion, normalization, retrieval, summarization, injection, and observation, each step governed by policy and each constrained by a finite budget. Context engineering is the discipline of designing that pipeline so that the tokens entering the model carry maximum useful signal per unit of cost and attention.

This is not prompting. Prompting assumes a small, fixed, user-authored string. It is not modeling. Modeling concerns the weights and the architecture. It is not training. Training is about gradients over corpora. Context engineering sits between training and inference: it shapes the inference-time substrate on which the model's parametric knowledge actually runs. The remainder of this book treats that substrate as a first-class engineering object — measurable, optimizable, and prone to specific, diagnosable failure modes.

1.1 Context as an Engineered Substrate

A foundation model at inference is a function from a token sequence to a token sequence. The function is fixed at deployment time; what varies is the sequence itself. That sequence is the model's *context window*, and everything that lands inside it is context. It is tempting to think of context as the user's message, possibly augmented with a system prompt. That view collapses the moment you put a model behind a real product. The context a production model sees typically contains at least seven distinct component classes:

1. System instructions — the persistent persona, format constraints, tool-use grammar, and policy guardrails. 2. Developer-supplied few-shot exemplars — demonstrations used to anchor style or reasoning patterns. 3. User turn — the immediate message, often itself preprocessed (PII redaction, intent classification, language detection). 4. Conversation history — prior turns, possibly compressed or truncated. 5. Retrieved documents — chunks pulled from a vector store, knowledge graph, or web index. 6. Tool traces — inputs and outputs of function calls, including intermediate reasoning. 7. Working state — scratchpads, plan objects, structured memories, session variables.

A common architectural mistake is to treat these seven as a flat string concatenated in arbitrary order. They are not interchangeable. Each has a different lifetime, a different source of authority, a different update cadence, and a different cost. The system instructions change rarely. The retrieved documents change every query. The tool traces are causal descendants of the model's own previous outputs. Conflating them into "context" obscures the engineering decisions that actually determine model behavior.

Consider a customer-support assistant that answers questions about a SaaS billing system. The naive design is:

``` SYSTEM: You are a helpful support agent. Be concise. USER: Why was I charged $47 last Tuesday? ```

A context-engineered design looks like:

``` SYSTEM: You are Tier-2 support for AcmeBilling. Persona: warm, precise. Never invent figures. If uncertain, escalate. Tool: lookup_invoice. DEVELOPER_EXAMPLES: <3 annotated turns demonstrating correct citation format> SESSION_STATE: {customer_id: 88421, plan: "Pro Annual", tenure_days: 412} RETRIEVED: <top-3 invoice records for customer 88421 in last 30 days> RETRIEVED: <excerpt from billing FAQ on prorated charges> TOOL_TRACE: lookup_invoice(customer=88421, date_range=last_30d) → 2 records USER: Why was I charged $47 last Tuesday? ```

Both blocks feed the same model and elicit the same general capability. The second is engineered: each component was selected by a policy, retrieved by a function, and placed intentionally. The first is *prompted*. The difference is not aesthetic. The second pipeline can be measured, versioned, A/B tested, debugged when it fails, and rolled back when a retrieval index corrupts. The first can only be rewritten by hand.

This reframe has three immediate consequences:

• **Context is a data structure, not a string.** It has typed fields, schemas, and invariants. Treating it as opaque text loses the ability to reason about staleness, provenance, and eviction.
• **Context is an output.** Every component in the window is the output of some upstream process — a retrieval function, a summarization model, a tool, a policy. The model does not "see" intent; it sees what the pipeline produced.
• **Context has a budget.** Tokens cost money, latency, and attention. Every component competes against every other for finite space. Engineering context is largely engineering that budget.

These three properties — structure, output, budget — define the substrate this book is about. The rest of this chapter names the disciplines that operate on that substrate, the failure modes that arise when they are ignored, the vocabulary that makes those failures discussable, and the reference architecture that ties them together.

1.2 The Four Pillars and Their Interaction Surface

Context engineering rests on four interlocking pillars. Each addresses a different constraint of the substrate; each has its own literature, tooling, and pitfalls; and none of them works in isolation. The pillars are:

1. Long context — operating with model context windows that, in current frontier systems, range from 32k to over 1M tokens. 2. Compression — reducing the token count of arbitrary content while preserving task-relevant information. 3. Summarization — producing a lossy but coherent narrative representation of a long input. 4. Retrieval and memory — selecting and incorporating external state on demand. 5. State management — tracking what the model knows, what it has forgotten, and what it should not yet see.

(Some treatments fold retrieval and memory together; here they are split, because the engineering questions differ: retrieval is a per-query decision; memory is a per-session or cross-session policy. State management is the umbrella under which both live.)

1.2.1 Long Context

Long-context capability is the load-bearing assumption of context engineering. If models could only see 4k tokens, the discipline would collapse to "be clever about the prompt." Frontier models can now attend over hundreds of thousands of tokens, and the engineering question has shifted from "what can we fit" to "what *should* we fit, and where."

Long context is not a free resource. Empirically, attention is not uniform: models attend more reliably to the beginning and end of a context (the primacy and recency bands) and degrade in the middle, a phenomenon Liu et al. (2023) called lost-in-the-middle. Subsequent work has shown that this is not merely a benchmark artifact: even on production workloads, retrieval accuracy and instruction-following degrade as relevant content is moved toward the center of the window. Long context also costs quadratically in self-attention and linearly in KV-cache memory; a 1M-token context can cost an order of magnitude more than a 32k one for the same query.

The engineering implication is that a longer window is not a license to stop curating. It is an invitation to design layout policies that exploit primacy and recency, and to instrument which regions of the window the model actually used.

1.2.2 Compression

Compression reduces token count without necessarily changing semantics. It encompasses:

• **Token-level compression**: replacing verbose phrasings with terser equivalents, normalizing whitespace, removing stopwords where the model tolerates them.
• **Structural compression**: collapsing JSON paths, dropping null fields, abbreviating repeated keys.
• **Re