Foundations of Large Language Models

Foundations of Large Language Models

The emergence of large language models (LLMs) represents one of the most significant technological developments in the history of artificial intelligence. These systems, capable of understanding and generating human language with remarkable proficiency, have fundamentally altered how we interact with machines and process information. Yet beneath their seemingly intuitive behavior lies a sophisticated architecture built upon decades of research in statistical modeling, neural networks, and computational linguistics. To master the art of prompt engineering—that is, the systematic craft of communicating effectively with these models—one must first develop a thorough understanding of how language models actually function.

This chapter establishes the theoretical and practical foundations necessary for all subsequent discussions of prompt engineering techniques. We begin by tracing the historical evolution of language models from their statistical origins through the revolutionary transformer architecture. Following this contextual framework, we examine the core mechanisms through which modern LLMs process and generate text, with particular attention to the attention mechanism that enables sophisticated contextual understanding. The concept of next-token prediction, which forms the fundamental operating principle of autoregressive language models, receives detailed treatment. We then introduce and define the key technical concepts—tokens, embeddings, and hidden states—that permeate all discussions of LLM behavior. Finally, we distinguish between different categories of language models, particularly the critical distinction between base models and instruction-tuned models, as this distinction has profound implications for how one should approach prompt design.

Understanding these fundamentals is not merely academic exercise; it directly informs effective prompting practice. When you understand that a language model generates text by predicting one token at a time based on probabilistic distributions learned during training, you begin to appreciate why certain phrasings elicit more coherent and useful responses than others. The theoretical knowledge presented in this chapter will illuminate the reasons behind the practical techniques explored throughout this book, transforming prompt engineering from an exercise in trial and error into a disciplined discipline grounded in computational principles.

The Historical Evolution of Language Models

The story of language modeling begins not with neural networks, but with probability theory and linguistics. Understanding this historical progression illuminates why modern language models take the form they do and provides essential context for comprehending their capabilities and limitations.

Early Statistical Approaches

The earliest language models were fundamentally statistical in nature, attempting to capture the regularities of language through probability distributions over sequences of words. The foundational concept here is the n-gram model, which predicts a word based on the preceding $n-1$ words. A bigram model ($n=2$), for example, would estimate the probability of the word "cat" following "the black" as $P(\text{cat}|\text{"the black"})$. Trigram models ($n=3$) consider two preceding words, and higher-order n-gram models capture longer-range dependencies.

These statistical models operated on a straightforward principle: patterns observed frequently in training data would be assigned higher probabilities, while rare or unseen combinations would receive lower probabilities. A bigram model trained on a corpus containing many instances of "The cat sat on the mat" would assign high probability to the sequence "cat sat," because this particular word pair appeared frequently. Conversely, the probability of "cat drove" would be vanishingly small unless such an unusual sequence appeared in training data.

The limitations of n-gram models became increasingly apparent as researchers attempted to build more capable systems. First, these models suffered from a fundamental curse of dimensionality: the number of possible word sequences grows exponentially with the sequence length, meaning that even massive training corpora could not adequately capture all but the most common patterns. Second, n-gram models had no mechanism for generalizing beyond exact sequences observed during training—they could not understand that "The large cat" and "The big cat" express similar concepts, because these specific word combinations might never have appeared together in training data. Third, capturing long-range dependencies required enormous values of $n$, which further exacerbated the data sparsity problem.

Despite these limitations, n-gram models established several crucial principles that remain relevant today. They demonstrated that language has statistical regularities that can be learned from corpora, that prediction is fundamentally a probabilistic process, and that models must balance specificity (capturing exact phrasings) with generalization (handling novel combinations). These insights would guide the development of more sophisticated approaches.

The Neural Network Revolution

The introduction of neural networks to language modeling represented a paradigm shift in how language patterns could be captured and exploited. Rather than calculating probabilities directly over discrete word sequences, neural language models learned continuous representations of words and used distributed computations to capture linguistic regularities.

The seminal work of Bengio et al. (2003) introduced the neural language model that would become the foundation for modern approaches. This model learned a distributed representation (now called an embedding) for each word in the vocabulary, mapping discrete word symbols to continuous vectors in a high-dimensional space. These embeddings captured semantic relationships: words with similar meanings or syntactic roles would be positioned close to each other in the embedding space. The model then used a feedforward neural network to predict the next word given the embeddings of preceding words.

This approach offered dramatic improvements over n-gram models. Because embeddings represented words as continuous vectors, the model could generalize from observed patterns to similar but unobserved combinations. If "big" and "large" had similar embedding vectors, the model could make reasonable predictions about "large cat" even if it had only observed "big cat" during training. The neural network component learned complex non-linear relationships between word sequences that simple co-occurrence statistics could not capture.

The subsequent development of recurrent neural networks (RNNs), particularly Long Short-Term Memory (LSTM) networks introduced by Hochreiter and Schmidhuber (1997), enabled language models to process sequences of arbitrary length. RNNs maintained a hidden state that was updated at each time step, theoretically allowing them to capture dependencies across arbitrary distances in the input sequence. In practice, however, training RNNs to capture long-range dependencies proved challenging due to problems of vanishing and exploding gradients.

The introduction of the attention mechanism by Bahdanau et al. (2014) represented a crucial advance, allowing models to directly attend to any position in the input sequence when making predictions. Rather than being forced to compress all relevant information into a fixed-size hidden state, models could selectively focus on the most relevant parts of the input. This mechanism would become the cornerstone of the transformer architecture that would revolutionize natural language processing.

The Transformer Revolution

The introduction of the transformer architecture by Vaswani et al. (2017) in their landmark paper "Attention Is All You Need" marked the beginning of a new era in language m