Foundations of Context: Defining the Representational Substrate

Foundations of Context: Defining the Representational Substrate

The term "context" performs a peculiar kind of conceptual work in LLM systems. In a single-turn completion, it appears to be a stable referent—the text preceding the model output, nothing more, nothing less. The moment one moves into a production application, that stability dissolves. The same word must refer simultaneously to the bytes physically consumed by a transformer, to the documents available in a retrieval index, to the JSON payload constructed by an orchestration layer, and to the implicit assumptions shared between developer and reviewer. Confusion among these referents is not a stylistic problem; it is a primary source of silent failures, billing surprises, and intractable production incidents. This chapter establishes a precise vocabulary for the rest of the book. It argues that the intuitive notion of context is in fact a stack of three distinct tiers, each with its own ownership, lifecycle, and failure modes; it then reframes context in information-theoretic terms as conditional information under uncertainty; it surfaces the empirical fact that advertised context windows are not the same as usable context; and it consolidates these insights into a single optimization problem—maximizing signal-to-noise ratio under a multi-dimensional budget. The chapter closes by previewing the four-pillar framework—long context, compression, retrieval memory, and state management—that organizes the engineering techniques developed throughout the remaining chapters.

1.1 The Three-Tier Taxonomy

Consider a deployed assistant answering a question about a customer's most recent order. The engineer's mental model includes, at minimum, the following elements: the user message typed into the chat box, the system prompt hidden from the user, a retrieved order record fetched from a transactional database, a snippet of the company's return policy retrieved by similarity search, a summary of the previous four turns in this conversation, a user profile containing loyalty tier and language preference, the current timestamp and timezone, and a list of available tools the model may invoke. Each of these elements is, in some defensible sense, "context." A single model invocation will not receive all of them in equal measure. Some will appear in the assembled prompt; some will be silently excluded by the orchestration logic; some will be present only as latent state that the engineer's code never actually transmits. The result is that "what context the model saw" and "what context existed in the system" can diverge by an order of magnitude and yet remain invisible to anyone reading the prompt in isolation. Production systems that are difficult to debug are almost always difficult to debug precisely because this divergence is unobserved.

To make the divergence explicit and actionable, the book introduces a three-tier taxonomy. Each tier corresponds to a different engineering abstraction with distinct ownership and observability characteristics.

Tier 1: Internal Model Context

The internal model context is the sequence of tokens that the transformer is actively attending to at inference time. It is bounded physically by the attention window—typically expressed in tokens—and operationally by the KV cache, positional embeddings, and any sliding-window or sparse-attention scheme the architecture employs. Tier 1 is the most concrete of the three tiers: it has a measurable byte length, a definite position in memory, and a direct causal effect on the model's next-token distribution.

Two properties of Tier 1 deserve emphasis. First, it is opaque to the application. The application does not directly construct the KV cache or the attention pattern; it constructs a sequence of tokens, which the runtime subsequently tokenizes, embeds, and caches. The application cannot, in general, predict which spans of its engineered context will receive the most attention from the model at generation time. This asymmetry between what the application thinks it is emphasizing and what the model actually attends to is the source of much of the surprise documented in prompt engineering folklore. A system prompt positioned at the beginning of the input does not necessarily receive more attention than a retrieved document positioned in the middle; the actual attention pattern depends on the model's training, the relative entropy of the spans, and the presence of structural cues such as role markers or section headers. Second, Tier 1 has a hard ceiling. Exceeding the advertised window produces a runtime error; in production, this often manifests as a 400 response, an empty completion, a model fall-through to a smaller fallback, or a silently truncated input. The hard ceiling is well understood and trivially measurable, which is precisely why the soft degradation discussed in Section 1.3 is so much more dangerous: it produces no signal at the boundary.

Tier 2: Application Context

The application context comprises everything available to the running system that is not yet encoded into a model input. This includes documents indexed for retrieval, results of tool calls executed in earlier turns, conversation histories persisted in a session store, user profiles, environmental configuration, and any other state the orchestrator can read. Tier 2 is, in principle, unbounded: a single application can index millions of documents and retain session histories indefinitely.

Tier 2 has its own internal structure that matters for engineering. A retrieval index is a function from a query to a ranked list of candidate documents; the index can be queried but not directly inspected. A conversation history is a sequence with structural metadata—turn indices, role labels, timestamps, and possibly trust annotations. A user profile is typically a structured record rather than free text. A tool registry is a set of callable functions with typed schemas and authorization scopes. Treating Tier 2 as an undifferentiated pile of "context" obscures these structural differences and leads to engines that query all of them with the same primitive, applying the same freshness assumptions, the same cost model, and the same reliability expectations to sources that in fact differ along all three dimensions.

A retrieval index, for instance, can be stale relative to a transactional database; the freshness lag is an engineering parameter that the orchestrator must reason about. A tool call result is point-in-time: it reflects the world at the moment of execution and decays in relevance thereafter. A user profile changes only on explicit user action. A session state is mutable within a session but persistent across sessions only if deliberately stored. Conflating these temporal characteristics produces systems that, for example, cite a policy that was superseded three weeks ago, or refer to a user preference the user explicitly revoked.

Tier 3: Engineered Context

The engineered context is the payload that the application constructs and transmits to the model. It is a function of Tier 2: an orchestrator selects some subset of available state, formats it into tokens, and assembles the result into the sequence that becomes Tier 1 in the next forward pass. Tier 3 is the explicit artifact of engineering work—every token in it was placed there by a deliberate decision, whether made by hand-written code, a templating engine, or an upstream model call.

This tier is where context engineering actually happens. The design questions—how much retrieval output to include, in what order, with what delimiters, against what fixed instructions—are all questions about Tier 3. The trade-offs—density versus completeness, freshness versus cost, instruction following versus recall—are all trade-offs evaluated against Tier 3. Treating Tier 3 as a transparent, frictionless interface between Tier 2 and Tier 1 is the most common conceptual error in the field. Tier 3 is not t