Foundations: Why Context Is the Substrate of Intelligence

Foundations: Why Context Is the Substrate of Intelligence

The reason your LLM system hallucinates, contradicts itself, ignores instructions, or spends $0.40 on a query that should cost $0.02 is almost never the model. It is the context. Specifically, it is the mismatch between what the model needs to know in order to produce the answer you want and what you actually put inside its context window before inference begins.

This claim is unfashionable. The dominant narrative of the past two years has been architectural: longer windows, smarter attention patterns, more parameters, better training. Each of those is real, but each is downstream of a more fundamental fact that engineers routinely ignore: every behavior of a large language model is downstream of what sits inside its context window at the moment of inference. The model does not have memory in any persistent sense. It does not know what it knew yesterday, what the user said five turns ago, or what the system prompt says unless that information is, right now, represented as tokens inside the buffer the attention mechanism can read.

If your mental model of context is "the prompt you happen to send," you will scale poorly, because adding more sources of context creates compounding interference rather than compounding capability. You will pay too much, because unused or low-value tokens still cost the same to process as high-value ones. You will ship unreliable products, because small changes in context ordering produce non-monotonic changes in output quality. And you will be unable to debug failures, because the relationship between your context and the model's behavior will look like noise.

This chapter fixes the mental model. By the end of it, you will have a formal definition of context, an understanding of why nominal capacity and effective capacity are not the same number, a vocabulary for the four forces that govern every token's contribution to a model's output, and a reusable diagnostic — the Context Inventory Worksheet — that you can apply to any LLM system you have already shipped or are about to ship. None of this is theoretical. Every concept maps to a knob you can adjust, and every knob has measurable consequences.

1.1 A Formal Definition of Context

For the rest of this book, context is defined as follows:

> Context is the finite, ordered, attention-addressable sequence of tokens present in the model's input buffer at the moment an inference step begins, together with any tokens the model has already generated during the current decoding session.

Four words in that definition carry the entire weight of this chapter: finite, ordered, attention-addressable, and tokens. Each one rules out a common misconception, and each one has consequences for how you engineer LLM systems.

1.1.1 Finite

The context buffer has a hard capacity bound. For current frontier models this bound ranges from roughly 8,000 tokens (smaller open-source variants) to several million tokens (long-context production endpoints), but it is always finite. The model does not have access to "the entire internet" or "your whole codebase" or "the full conversation history." It has access to whatever fits inside the buffer, and nothing else.

This seems obvious in the abstract, but in practice engineers violate it constantly in their mental models. Consider a Retrieval-Augmented Generation system that retrieves "the top 5 most relevant documents" and stuffs them into the prompt. If the top 5 documents together contain 40,000 tokens and your window is 32,000 tokens, you have not retrieved "the top 5 most relevant documents." You have retrieved a partial, truncated, possibly incoherent fragment of the top 5 most relevant documents, and the truncation itself is almost certainly not aligned with semantic relevance. The model is then asked to answer a question about documents it has not fully seen.

Capacity is the first constraint. Every other constraint in this chapter operates underneath it. Once you internalize that capacity is bounded, the question stops being "what should I include?" and starts being "given the bound, what is the best subset I can include?" — a question that has a definite answer for each task and each cost budget.

1.1.2 Ordered

The buffer is a sequence, not a set. Position matters. Tokens at the beginning of the buffer and tokens at the end of the buffer are not equivalent inputs to the model, even when their textual content is identical. The attention mechanism is position-sensitive in a way that humans rarely intuit, and the model has learned, during training, that different positions in the buffer carry different implicit roles (system instructions tend to appear at the start; user messages tend to appear near the end; assistant turns interleave with user turns).

Two consequences follow:

1. Reordering changes output. Move a system prompt from the top of the buffer to the bottom and you will, in expectation, change the model's behavior. Move a piece of evidence from the middle of a retrieved-document block to the top of that block and you will change how much weight that evidence carries. This is not a bug; it is the architecture. 2. Recency is a force, not a preference. Tokens that appeared recently in the buffer are easier for the model to attend to than tokens that appeared long ago, all else equal. This is the recency force discussed in §1.2.

Order is one of the cheapest levers you have. You can change it without changing any other component of your system, and the resulting behavioral delta is often larger than the delta from changing models. Many production systems, audited carefully, reveal that their prompt ordering was set once, years ago, by the first engineer who wrote the template, and has not been re-evaluated since.

1.1.3 Attention-Addressable

This is the part most engineers skip, and it is the most important part. "Attention-addressable" means: whether the model can use a token depends on whether the attention mechanism will route information through that token to the output. A token can be physically present in the buffer and still be functionally invisible to the model — present but unattended.

Attention is a learned, query-conditioned operation. At each decoding step, the model constructs queries from the current decoding state and uses them to weight the keys derived from earlier tokens. A token's contribution to the current output is the soft-maxed compatibility of its key with the current query:

$$\alpha_i = \frac{\exp(q \cdot k_i / \sqrt{d})}{\sum_j \exp(q \cdot k_j / \sqrt{d})}$$

Tokens whose keys are incompatible with the current query receive effectively zero attention weight and contribute effectively zero information. The denominator sums over all tokens in the buffer, so the total attention budget is exactly 1. Adding more tokens does not add more attention; it dilutes the attention available per token.

This has three practical implications:

1. Inclusion is not influence. Just because a token is in the buffer does not mean it influences the output. Many tokens in any non-trivial prompt are, at any given generation step, contributing nothing. A prompt of 4,000 tokens might have only 200 of them doing real work at any given decoding step. 2. Salience is dynamic. Whether a token is attended to depends on what is currently being generated. A token that strongly influences one output step may have zero influence on the next. The model's effective context shifts with the decoding trajectory. 3. Attention is a budget. As noted, the total attention weight sums to 1 across all tokens. This is why "just add more context" is usually a regression, a topic we take up in §1.3.

The phrase "attention-addressable" exists to break the illusion that the buffer is a place where information is uniformly available. It is not. It is a place where information is conditionally available, and the conditions are gover