Foundations of Large Language Models and the Art of Prompting

The single most consequential fact about prompt engineering is also the one most newcomers underestimate: a prompt is not a query, it is a configuration vector. Every word, every whitespace token, every example you include is read by a transformer as a conditional input to a next-token probability distribution, and that distribution is the joint product of tokenization, learned attention patterns, an autoregressive decoder, and a multi-stage training pipeline that culminates in reinforcement learning from human feedback. To craft prompts that generalize, that are robust to paraphrase, that minimize cost, and that elicit structured reasoning rather than flattery, you must understand this pipeline end to end. This chapter builds that mental model from first principles, using only as much mathematics as is necessary to make architectural decisions feel inevitable rather than arbitrary, and ends with a working evaluation harness you can reuse for every experiment in the rest of the book.

Transformer Architecture Fundamentals: Tokenization, Attention, and Autoregressive Decoding

A modern large language model is, at the hardware level, a stack of identical transformer blocks, each performing two operations: a multi-head self-attention computation followed by a position-wise feed-forward network, with residual connections and layer normalization wrapping both. The input to this stack is not characters and not words but a sequence of integer token IDs, and the output at every position is a probability distribution over the vocabulary for the next token. Everything that prompt engineering can influence happens at the boundaries of this loop: the tokens we choose to put into the prompt, the way those tokens are embedded and attended to, and the way the model's logits are sampled back into text.

Tokenization Is the First Decision You Make, Whether You Know It or Not

Before a single attention head fires, the raw string of your prompt is sliced into tokens by a deterministic algorithm. The dominant family of algorithms in 2024–2026 is Byte Pair Encoding (BPE) or one of its variants (WordPiece, SentencePiece, tiktoken's BPE, GPT-4o's o200k_base). BPE is a greedy bottom-up merger: starting from a base vocabulary of individual bytes or Unicode code points, the algorithm iteratively merges the most frequent adjacent pair into a new symbol until the target vocabulary size is reached. The crucial consequence is that the mapping from string to token sequence is lossy and uneven: a common English word like "the" is one token, a rare technical term like "polypharmacology" might be five, and the same logical word can tokenize differently depending on whether it has a leading space, trailing punctuation, or capitalization.

```python

Demonstrating tokenization variance with tiktoken (OpenAI's tokenizer)

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

samples = [ "The patient was diagnosed with polypharmacology concerns.", "The patient was diagnosed with polypharmacology concerns", # no period "the patient was diagnosed with polypharmacology concerns.", # lowercase " patient polypharmacology", # whitespace trick ]

for s in samples: ids = enc.encode(s) tokens = [enc.decode([i]) for i in ids] print(f"{len(ids):3d} tokens | {tokens}") ```

When you run this snippet you will see token counts of roughly 9–11 for sentences that differ by a single character. This matters because every prompt you write is billed in tokens, every token costs the same amount of attention compute, and a prompt that produces 30% more tokens than necessary is a prompt that is 30% more expensive at inference and that "uses up" 30% more of the model's finite context window. Worse, subtle tokenization collisions can cause silent prompt injection: the string `" prompt"` (with a leading space) and `"prompt"` (without) often fall on different token boundaries, which means that a sentence you copy-pasted from documentation may tokenize into a sequence the model has never seen during training, even if the human-readable string is identical.

A practical rule of thumb that holds across the major model families: assume that one English token corresponds to roughly 0.75 words, that code tokens are roughly 1.5–3× denser (less efficient) than English prose, that non-Latin scripts often tokenize into 2–5× more tokens per character, and that any prompt with significant whitespace, emoji, or non-ASCII characters should be re-tokenized before shipping.

Embeddings Turn Discrete IDs into a Geometric Space

Once tokenized, each integer ID is mapped by a learned embedding matrix $E \in \mathbb{R}^{V \times d}$ (where $V$ is vocabulary size, $d$ is the model's hidden dimension, typically 4096–12288 for current flagship models) into a continuous vector. A positional encoding is added—either sinusoidal as in the original Transformer paper, or, more commonly today, Rotary Position Embeddings (RoPE) applied inside the attention computation itself. RoPE encodes position by rotating each query and key vector by an angle proportional to its absolute position:

$$q_i' = R_{\theta, i} q_i, \quad k_j' = R_{\theta, j} k_j$$

where $R_{\theta, i}$ is a block-diagonal rotation matrix. The dot product $\langle q_i', k_j' \rangle$ then depends only on the relative offset $i - j$, which gives the model a translation-invariant notion of distance and, importantly, allows the trained model to extrapolate somewhat beyond its training context length (though not without quality degradation, which we will quantify in the economics section).

The geometric structure of the embedding space is the hidden substrate of every prompting trick in this book. Few-shot examples work because the model has learned a metric space in which semantically similar prompts cluster; chain-of-thought works because the autoregressive decoder can be steered through a sequence of intermediate embedding states that correspond to reasoning steps; instruction following works because the instruction-tuning phase has pushed the embedding trajectories of "imperative" sentences toward regions of activation space that produce compliant completions. None of these behaviors are emergent from magic—they are learned from data and constrained by architecture.

Attention Is the Working Memory of the Model

The transformer block's core operation is scaled dot-product attention. For each layer, the input sequence is projected into three matrices: queries $Q \in \mathbb{R}^{n \times d_k}$, keys $K \in \mathbb{R}^{n \times d_k}$, and values $V \in \mathbb{R}^{n \times d_v}$, where $n$ is the current sequence length. The attention output at position $i$ is a weighted sum of all value vectors, with weights computed as:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right) V$$

where $M$ is a causal mask—an upper-triangular matrix of $-\infty$ entries that prevents any position from attending to future positions. Multi-head attention runs $h$ such computations in parallel with different learned projections, then concatenates and projects the result. A typical modern model has 32–128 attention heads per layer and 80–120 layers, giving hundreds of distinct attention patterns that the model has learned to specialize: some heads track syntactic agreement, others copy rare identifiers, others route information between distant parts of a long document.

For prompt engineers, three properties of attention are first-order important.

First, attention is quadratic in sequence length. Doubling the prompt from 4,000 to 8,000 tokens quadruples the cost of every attention operation and roughly quadruples inference latency on most serving stacks. This is the primary reason "context windows are free" is a marketing fiction; you pay for them in attention compute, in KV-cache memory, and in latency.

Second, attention is content-addressed, not position-addressed. A token at position 1,500 that