Foundations of Context Engineering

Foundations of Context Engineering

When developers first work with large language models, they approach context as something simple: you provide a prompt, you get a response. This mental model works adequately for toy examples and single-turn interactions. But the moment you build anything resembling a real application—a customer service system, a code assistant, a document analysis tool—you encounter a fundamental challenge that simple prompt engineering cannot solve: the information environment that surrounds an LLM is not a fixed container but a dynamic, expensive, and often unpredictable resource that determines whether your application succeeds or fails.

The challenge manifests in concrete problems that every LLM application developer eventually faces. Your retrieval-augmented generation system retrieves relevant documents, but the model ignores the most important one because it appears in the middle of a long context. Your agentic workflow passes state between steps, but the model "forgets" critical constraints you established earlier. Your document summarization pipeline processes thousands of pages, but costs spiral out of control because you're sending everything to the model when only a fraction is relevant. These are not isolated bugs with isolated solutions—they are symptoms of a deeper discipline gap that context engineering exists to fill.

This chapter establishes the conceptual foundations for that discipline. We begin by defining what context actually means in the LLM context, distinguishing it from the narrow notion of "input text" that misleads many practitioners. We then trace the historical evolution from simple prompt engineering to the sophisticated context manipulation techniques that modern applications require. The core tension we examine—between context richness and context manageability—structures everything that follows. We introduce the three-pillar framework that organizes this entire book: context manipulation, context retrieval, and context persistence. Finally, we provide a practical methodology for auditing your own applications to identify context-related bottlenecks and optimization opportunities.

By the end of this chapter, you will understand why context engineering has emerged as a distinct discipline, what fundamental challenges it addresses, and how the three pillars of manipulation, retrieval, and persistence interact to form a coherent architecture for building reliable, scalable, and cost-effective LLM applications.

The Expanding Definition of Context

The word "context" appears constantly in LLM discussions, but its meaning is often assumed rather than defined. This casual usage obscures a concept that is both more powerful and more complex than most practitioners initially appreciate. Understanding the true scope of context is the first step toward engineering it effectively.

Context is not merely the input text to a model—it is the entire information environment that influences model behavior. When you send a request to an LLM, the model does not process your text in isolation. It processes a constructed sequence that includes your prompt, but also system-level instructions, examples of desired behavior, relevant documents retrieved from external sources, the history of the current conversation, and various implicit signals about formatting, tone, and task framing. Each of these elements shapes the model's output, and the way they interact—with each other and with the model's internal knowledge—determines whether your application delivers value or frustration.

This distinction matters because it reframes the engineering challenge. If context were simply "what you type," there would be little to engineer beyond writing better prompts. But because context encompasses everything that informs the model's response, every decision about what to include, how to structure it, and how to manage it across interactions becomes an engineering decision with concrete consequences.

Consider the practical implications. A code generation assistant needs not just the user's current request but also the relevant portions of their codebase, documentation about the libraries they use, examples of similar solutions, and perhaps style guidelines established earlier in the conversation. A legal document analyzer needs the full text of contracts under review, relevant case law, internal firm guidelines, and templates for the type of analysis requested. An educational tutor needs the student's history of questions and misconceptions, the material being studied, pedagogical strategies appropriate to the learner's level, and feedback from previous sessions.

In each case, the "prompt" that the user types is the smallest and least significant portion of the actual context the model processes. The bulk of the context is constructed, retrieved, formatted, and managed by the application—this is where context engineering lives.

The Cost of Unbounded Context

Understanding context as the entire information environment immediately reveals why long context windows, while seemingly a solution, create new categories of problems. Early language models had context windows of 2,000-4,000 tokens—enough for a few paragraphs of text but nothing more. Developers adapted by keeping prompts short, extracting only the most essential information, and designing applications around severe context constraints.

The arrival of models with 100K, 200K, and even million-token context windows appeared to remove these constraints entirely. Why carefully select relevant documents when you could simply include everything? Why worry about conversation history when you could stuff years of dialogue into a single request? The logical answer seemed obvious: context limits were no longer a meaningful constraint, and developers could finally focus on building rich, information-dense applications.

The reality proved more complicated. Longer context windows did not eliminate the challenges of context management—they transformed them. The fundamental tension that emerged is between context richness and context manageability.

More context generally leads to better model responses. When a model has access to relevant information, it can provide more accurate, more detailed, and more appropriate answers. A legal assistant with access to all relevant case law and statutes produces better analysis than one working from memory alone. A code assistant with the full repository context can generate more correct and consistent code than one working from snippets.

However, unbounded context creates cascading practical problems:

Increased latency. Processing longer contexts requires more computation. A request with 100K tokens takes substantially longer to generate a response than one with 10K tokens, not just because of the input processing but because attention mechanisms scale quadratically with sequence length. For applications where responsiveness matters—customer service, real-time assistance, interactive tools—unbounded context can make the difference between a usable product and an abandoned one.

Escalating costs. Token counts directly translate to costs in most LLM APIs. Beyond the direct cost of processing longer inputs, longer contexts often lead to longer outputs, compounding the expense. An application that naively includes everything "just in case" can easily run 10-100x the cost of a carefully optimized version that includes only what is necessary.

Degraded attention quality. The famous "lost in the middle" problem reveals that models do not attend uniformly to all parts of a long context. Information placed in the middle of a lengthy sequence receives less attention than information at the beginning or end. Important details can be overlooked simply because of their position, not because they are irrelevant or poorly expressed. As context windows grow, this problem intensifies—the middle of a 100K token sequence is a v