Foundations of Large Language Models and the Prompt Interface

Foundations of Large Language Models and the Prompt Interface

What does a prompt actually *do* to a transformer, and why is prompt engineering a form of program synthesis over a learned distribution? The answer is not "it asks the model to do something." That framing imports a folk-psychology of intent that has no analogue in the math. A more useful answer begins with a single equation: at inference time, a transformer defines a probability distribution $P(y \mid x)$ over token sequences, where $x$ is the prompt and $y$ is the continuation. Everything else — the chat template, the system prompt, the few-shot examples, the chain-of-thought scaffolding — is an *intervention on the conditioning context* designed to bend that distribution toward a region the engineer wants sampled. This chapter is the formal substrate on which every subsequent chapter rests. We will reconcile the prompt with the autoregressive objective, dissect the transformer block at a level sufficient to reason about what a model can and cannot extract from a context window, expose tokenization as a first-class design variable rather than a preprocessing detail, distinguish the three prompting regimes carved out by pre-training, instruction-tuning, and reinforcement learning from human feedback (RLHF), and close by building a small instrumentation harness that lets the reader observe, rather than assume, that prompt structure matters.

1. The Autoregressive Substrate

A transformer language model is, in its bones, a next-token distribution estimator. Given a sequence of tokens $x_1, x_2, \dots, x_n$, it produces a categorical distribution over the vocabulary $\mathcal{V}$ at position $n+1$:

$$P_{\theta}(y \mid x_{1:n}) = \frac{\exp\left(\ell_y\right)}{\sum_{v \in \mathcal{V}} \exp\left(\ell_v\right)}$$

where $\ell = W_{\text{lm-head}} \cdot h_n$ is the logit vector obtained by projecting the final hidden state $h_n$ through the language modeling head. The pre-training objective that shaped $\theta$ is the negative log-likelihood summed across all positions:

$$\mathcal{L}_{\text{LM}}(\theta) = -\sum_{t=1}^{T} \log P_{\theta}\!\left(x_t \mid x_{<t}\right)$$

This is the only inductive bias the model was *directly* trained on. The vast majority of behavior people attribute to "instruction following," "reasoning," or "personality" is a byproduct of this objective applied to an extremely large, carefully curated corpus.

1.1 Prompting is conditioning, not commanding

When an engineer writes a prompt, they are not issuing a command to an agent. They are constructing a string $x$ that, after tokenization, becomes a prefix over which the model computes $P_\theta(\cdot \mid x)$. The action of decoding — greedy, nucleus, top-$k$, beam search — is then a constrained sampler over that conditional distribution. So the categorical layout of the prompt determines which continuations are *likely*, not which continuations are *intended*.

This reframing has three immediate consequences.

1. Soft constraints, not hard instructions. The model can — and frequently does — generate continuations that contradict the prompt. Pre-training data contains examples where prompts are *not* followed, and the model has internalized that pattern. A prompt that says "be concise" is a statistical nudge, not a syntax rule.

2. Compositional pressure from the corpus. Every property the prompt leverages must be discoverable in the training distribution. If the prompt elicits behavior, it is because some analogous surface form existed in the data. Novel combinations of words can work precisely because the model has learned factorized representations that generalize across combinations.

3. The chat template is part of the prompt. In production systems, the user-visible text is wrapped in a templated format (`<|user|>...<|assistant|>`, ChatML, Llama-3 chat markup, etc.). The model never sees raw user text in isolation; it sees templated text. This means prompt engineering is partly a *format engineering* problem — the same words in different templates produce different continuations because the model has been instruction-tuned on the templated distribution.

This reframing is not merely philosophical. It is the foundation that allows the rest of the book to treat prompt engineering as a rigorous optimization problem: define an objective over continuations, characterize the support of $P_\theta$, and search the discrete prompt space for prefixes that maximize (or constrain) the sampled output.

1.2 Pre-training as a compression of human knowledge

There is a useful, if slightly reductive, way to think about the pre-training phase: it is lossy compression of a large fraction of the public internet, of code repositories, of books, and of structured reference material. The transformer is *not* a database; it cannot look up "the third word of paragraph 7 of document X" reliably unless that exact sequence was duplicated many times during training. It *is*, however, a high-fidelity model of local and medium-range statistical dependencies, with weaker but non-trivial long-range dependencies mediated by attention.

A single number illustrates the scale. A 70-billion-parameter model with vocabulary size $V = 100{,}000$ and context length $L = 8{,}192$ specifies, at every decoding step, a categorical distribution over 100,000 outcomes. The number of distinct "state machines" the model can implement is intractable to enumerate — and that is precisely why prompt engineering remains a non-trivial research-grade activity rather than a solved lookup problem. The search space is enormous, the objective is poorly defined, and the evaluation is expensive. These properties are shared with classic program synthesis, and we will return to that analogy at the end of the chapter.

1.3 Sampling as the operational semantics

It is tempting to treat the model's *top-1* token as the model's "answer." This is the source of countless misunderstandings. Sampling parameters — temperature $\tau$, top-$p$ (nucleus) threshold, top-$k$, repetition penalty, frequency penalty, logit bias, mirostat — are *part of the inference contract*. They are not post-hoc tweaks. Two different decoding strategies from the same prompt can yield qualitatively different behaviors: greedy decoding tends to degenerate into loops; high-temperature sampling collapses coherence; nucleus sampling balances diversity and fluency. A practitioner who has not measured their prompt at multiple sampling settings has not measured the prompt at all.

Operationally, the formal description of nucleus sampling is

$$V^{(p)} = \min \left\{ v^{(1)}, v^{(2)}, \dots \in \mathcal{V}_{\text{top}} \;:\; \sum_i P(v^{(i)} \mid x) \ge p \right\}$$

followed by renormalization and categorical sampling from $V^{(p)}$. Temperature enters by scaling logits:

$$P_\tau(y \mid x) = \frac{\exp(\ell_y / \tau)}{\sum_v \exp(\ell_v / \tau)}$$

with $\tau \to 0$ recovering greedy decoding. These formulas are not aesthetic; they are the levers by which a prompt's *intended* distribution is transformed into the actual distribution over completions. We will return to them in Chapter 5 when we discuss decoding-aware prompt design.

2. Attention Topology and the Limits of In-Context Information Aggregation

Prompt engineering folklore is rich with techniques that amount to *rearranging tokens* to coax the model toward a behavior: putting the instruction before the context, repeating the question, swapping the order of few-shot examples, appending "Let's think step by step." Some of these matter enormously; some are superstition. Distinguishing them requires understanding what attention can and cannot do, which requires a brief, but careful, walk through the transformer block.

2.1 The transformer block, in just enough detail

For a sequence of token embeddings $E \in \mathbb{R}^{n \times d_{\text{model}}}$, a single decoder block computes

$$\begin{aligned} Q &= E W_Q, \quad K = E