What Is Context, and Why Does AI Need It?

The Moment an AI Forgets

Picture this. You are texting a friend about a tricky math problem. You explain the setup. They ask a clarifying question. You answer. You go back and forth for ten minutes, building up the conversation piece by piece. Then you write, "So if I substitute x = 3, what happens next?" and they reply, "Wait, what are we working on?"

You would be confused. Maybe a little annoyed. You might say, "We were solving the quadratic. I told you this ten minutes ago. We are on step four."

Your friend, to be fair, might have been distracted, or juggling a dozen other conversations, or genuinely just lost the thread. We have all done it. Human memory is fragile.

Now imagine the same thing happening to a computer program. Not because it was distracted. Not because it had other conversations going on. But because the way it processes information has a hard, physical limit, and once that limit is reached, the oldest information simply disappears. No warning. No error message. Just a polite response that quietly proves it has no idea what you were talking about.

This is the everyday reality of large language models. These are the AI systems that power chatbots, coding assistants, and writing tools. They are extraordinary. They can translate languages, write essays, debug code, and explain quantum physics. But they have a peculiar kind of memory that is nothing like yours and nothing like the memory of a traditional computer. It is bounded, expensive, and sometimes surprisingly fragile.

To build anything useful with these AI systems, to make them feel less like a distracted friend and more like a reliable partner, you have to understand a single concept above all others: context.

Context is the information an AI has "in mind" at the moment it generates a response. It is the working memory of a mind that, in some sense, does not exist between conversations. Every time you send a message, the AI wakes up, reads everything in its working memory, thinks about it, and replies. Then it goes back to sleep. When you message it again, it has to be reminded of who you are, what you wanted, and what just happened.

That reminder is context. And it is the most precious resource you have when working with AI.

This book is about the discipline of using that resource well. We call it context engineering. It is the practice of carefully deciding what information to put in front of an AI, in what order, in what form, and for how long, so that the AI can do its job without running out of the one thing it absolutely needs to think.

You will learn five core tools that context engineers use every day: long-context handling, compression, summarization, retrieval, and state management. Each is a way of solving a different kind of memory problem. Each has trade-offs. Each will be familiar by the end of this book.

But first, you need a clear picture of what context is, why it is always limited, and why throwing more of it at an AI is not always the right move.

Imagine you have just built a small chatbot to help students review for a history test. You give it a name. Let's call it Mia. You write a system prompt that says, "You are a friendly study buddy who helps high schoolers remember dates and events. Be encouraging and clear." You launch it. A student types, "Hi, I'm Alex. Can you quiz me on World War II?" Mia cheerfully agrees and asks the first question. Alex answers. Mia replies. The conversation goes on for fifteen minutes. Alex is learning, laughing, and feeling good.

Then, deep into the conversation, Alex writes: "I keep getting the date of D-Day wrong. Can you explain it one more time?"

And Mia says: "Of course! What would you like me to explain?"

Alex, slightly confused, writes: "D-Day. World War II. The beach landing. You were quizzing me on it ten minutes ago."

Mia: "I'm sorry, I'm not sure I have the context for that. Could you give me more details?"

The conversation has just broken. Not because Mia is broken, but because somewhere between the fifteenth question and the sixteenth, the earliest parts of the conversation—where Alex introduced themselves, where the topic was set, where the system instructions lived—slipped out of the AI's working memory. The context window, that bounded space where context lives, simply ran out of room.

By the end of this chapter, you will be able to explain exactly why that happened, why it had to happen, and why the fix is more interesting than "just give the AI more memory." Because in the world of AI, more is not always better. Smarter is better. Curated is better. Engineered is better.

Context Is the AI's Working Memory, and It Is Always Limited

To understand context, it helps to compare it to things you already know.

Think about a classroom. The teacher stands at the front and has a blackboard, or nowadays, a whiteboard or smartboard. Whatever they write on that board is what everyone in the room can see. It is the working space for the lesson. If the board fills up, the teacher has to erase something to make room for new writing. They might erase a worked-out problem from earlier in the period because the class has moved on to a new topic. The students who were not paying attention might miss what was erased, but the teacher still has the gist in their head.

A language model is a bit like that teacher, but with a crucial difference: the teacher remembers what was on the board even after erasing it. The AI does not. Once something leaves the AI's working memory, it is gone, and the only way for it to "know" that information again is for someone to put it back.

The "blackboard" of an AI is called the context window. It is a fixed-size buffer, measured in tokens, where the model keeps everything it needs to think about your next response. A token is the AI's unit of text. It is not exactly a word and not exactly a character, but somewhere in between. A rough rule of thumb for English: one token is about four characters, or about three-quarters of a word. The sentence "Context engineering is fun." is eight tokens. A typical page of a novel is around 500 tokens. A full-length book is somewhere between 80,000 and 120,000 tokens.

When you send a message to an AI, four kinds of things get added to the context window:

1. The system prompt. This is the instruction from the developer. It might say, "You are a careful, friendly assistant. Always explain your reasoning." It is the AI's "job description" and is usually placed at the very top of the context. 2. The conversation history. Every previous message between you and the AI, in order. 3. Your new message. The thing you just typed. 4. Any extra information the developer injected. This could be the text of a document, the output of a tool, a search result, or a structured note. We will see a lot of this in later chapters.

All of these get concatenated into a single stream of text. The model reads the entire stream from beginning to end, and based on everything it sees, it generates its next response. The response itself is also added to the context window, so that the next time you speak, the AI sees its own previous words as part of the conversation.

This is the first key idea: the AI does not have hidden memory. What it knows about your conversation is exactly what is in the context window at the moment it generates a reply. Nothing more.

Now, the size of that window is fixed. Different models offer different sizes, but they all have a hard limit. As of this writing, common context window sizes are:

• **8,192 tokens (8K).** Small but cheap. Good for short tasks.
• **32,768 tokens (32K).** A comfortable middle ground. Fits a short story or a long document.
• **128,000 tokens (128K).** A long context. Fits a small book.
• **200,000 tokens (200K) and beyond.** A very long context. Fits multiple books, hours of transcript, or a large codebase.

Newer models push the boundary further every year. Some experimental sy