LLM Mechanics 101 for Expert Prompt Engineers: What’s Actually Happening Under the Hood When You Type a Prompt

Last quarter, I was consulting for a SaaS company building a customer support chatbot powered by GPT-4. The team had spent weeks perfecting their prompt template, adding every possible edge case and instruction they could think of — the final prompt was 3,200 words long, and they were convinced it would eliminate 90% of support tickets. But when they launched, the chatbot missed 40% of the questions it was supposed to answer,

Last quarter, I was consulting for a SaaS company building a customer support chatbot powered by GPT-4. The team had spent weeks perfecting their prompt template, adding every possible edge case and instruction they could think of — the final prompt was 3,200 words long, and they were convinced it would eliminate 90% of support tickets. But when they launched, the chatbot missed 40% of the questions it was supposed to answer, and the most egregious failures were for the exact questions they’d spent the most time writing rules for: refund requests, account cancellations, and data exports. When I asked to see their prompt, the problem was obvious within 10 seconds: they’d buried their highest-priority instructions under 1,200 words of redundant company background, scattered unmarked scenario rules across the middle of the prompt, and tucked the core priority logic at the very end. They’d made every classic mistake that comes from treating prompt engineering as a game of throwing more text at the model, rather than understanding how LLMs actually process what you type.

This chapter is for the prompt engineers who are tired of hitting mysterious walls with their prompts, who’ve tried every template and trick from social media only to see inconsistent results. We’re not going to waste time rehashing “write clear instructions” or “use delimiters” — those are surface-level tips that only work if you understand why they work. Instead, we’ll break down the three core mechanical processes that happen the second you hit enter on a prompt: tokenization, attention mechanisms, and context window limitations. We’ll use plain, casual language, no oversimplified analogies that break down under scrutiny, and real examples from production systems to show how small changes to your prompt’s structure directly map to changes in how the model processes your input and generates output. We’ll also debunk the most pervasive myths about LLM “reasoning” that lead to bad prompt design, and end with hands-on exercises you can run against any open or closed-source model to test your understanding.

Tokenization: The First Filter Your Prompt Has to Survive

Before your prompt ever reaches the neural network that powers your LLM, it has to pass through the tokenizer — the often-overlooked first step in the entire pipeline. Most prompt engineers think of LLMs as processing words, or even characters, but that’s not how they work at all. Every input (and output) is split into discrete units called tokens, which are the only things the model can actually process. Think of tokens as the Lego bricks the model uses to build its understanding of your prompt: if you give it the wrong bricks, or too many useless bricks, it’s never going to build the structure you want.

What Tokens Actually Are (And Why They’re Not Words)

Tokens are not words, and they’re not characters. They’re statistically derived subword units generated by an algorithm called Byte Pair Encoding (BPE), first proposed by Google in 2016 for neural machine translation, and now used by every major LLM from GPT-4 to Llama 3 to Claude 3. BPE works by analyzing a massive corpus of training text to find the most common character sequences, and merging them into single tokens. The more common a sequence of characters is in the training data, the more likely it is to be a single token; rare sequences get split into multiple smaller tokens.

For English text, the average token is roughly