What Are Language Models? A Gentle Introduction

What Are Language Models? A Gentle Introduction

Consider this familiar scenario: You're typing a message to a friend on your phone, and before you finish the word "tomorrow," three suggestions appear—"morning," "afternoon," and "evening." Your phone seems to know what you want to say next. Or perhaps you've noticed how your email application sometimes finishes your sentences for you, offering words and phrases that match your writing style. These seemingly magical moments of digital prescience aren't magic at all—they're the work of language models, sophisticated mathematical systems that have quietly become an integral part of our daily digital interactions.

Language models are, at their core, systems designed to predict the probability of sequences of words (or more precisely, sequences of tokens, which we'll explore shortly). When your phone's keyboard suggests the next word in your sentence, it's calculating which word is most likely to follow what you've already typed. When your email client auto-completes your thoughts, it's drawing on learned patterns of how words tend to appear together. These predictions may seem simple on the surface, but they represent one of the most profound achievements in the history of artificial intelligence—the creation of systems that can understand, generate, and manipulate human language.

This book takes you on a journey through the architecture of intelligence, exploring how these remarkable systems work from the ground up. We'll start with the fundamental concepts that underlie all language models, gradually building toward the sophisticated neural architectures that power today's most advanced systems. By the end of this chapter, you'll have a solid foundation for understanding what language models are, how they evolved, and why they represent such a significant milestone in the development of artificial intelligence.

The Fundamental Task: Predicting the Next Word

To understand language models, we need to start with a deceptively simple question: how do we decide what word comes next in a sentence? When you say "I am going to the ___," your brain immediately considers possibilities—store, park, office, beach. You don't consciously calculate probabilities, but your mind is performing a probabilistic assessment of what word fits best in that context. This is the fundamental task that language models automate.

A language model is essentially a probability distribution over sequences of words. Formally, if we denote a sequence of words as $W = (w_1, w_2, w_3, ..., w_n)$, a language model assigns a probability $P(W)$ to this entire sequence. More usefully for prediction tasks, it also calculates the conditional probability of the next word given all previous words:

$$P(w_{n+1} | w_1, w_2, ..., w_n)$$

This notation represents the probability that word $w_{n+1}$ follows the sequence of words $w_1$ through $w_n$. When your phone suggests "morning" after you've typed "good," it's calculating something like $P(\text{morning} | \text{good})$ and comparing it to the probabilities of other possible continuations.

The elegance of this approach lies in its simplicity. Rather than trying to encode explicit rules about grammar, meaning, or context, we can learn the patterns of language directly from examples. Show a language model millions of sentences, and it will gradually learn that "good morning" appears frequently while "good shoe" almost never does. The model doesn't understand what "morning" means in the way humans do—it simply knows that certain word sequences are more probable than others based on its training data.

This probabilistic foundation distinguishes language models from earlier approaches to natural language processing, which often relied on hand-crafted grammatical rules. Those rule-based systems required linguists to explicitly encode knowledge about subjects, verbs, objects, and all the intricate ways English (or other languages) combine them. Such systems could handle simple cases but struggled with the infinite variety and subtle patterns of natural language. Language models, by contrast, learn these patterns automatically from data, making them far more flexible and capable of handling the messiness of real-world language use.

Why Prediction Matters

You might wonder why prediction—the seemingly simple task of guessing the next word—leads to such powerful systems. The answer lies in what prediction requires. To predict the next word accurately, a system must understand grammar (so it knows which words can grammatically follow), semantics (so it knows which words make sense in context), pragmatics (so it understands the intended meaning), and world knowledge (so it can reason about what typically happens in various situations).

Consider the sentence: "After the rain stopped, the children went outside to ___." A good prediction here requires understanding that rain creates wet conditions, that children typically enjoy playing outside after rain, and that puddles or mud are common in such situations. The word "puddles" or "play" would be reasonable predictions, while "asteroids" or "teleport" would not, even though grammatically anything could technically follow the infinitive "to."

This means that to predict well across diverse contexts, a language model must develop internal representations that capture enormous amounts of world knowledge, grammatical structure, and contextual understanding. These rich representations emerge naturally from training on large amounts of text—the model learns to predict words, and in doing so, it must implicitly learn everything necessary to make good predictions. The power of language models comes not from the prediction task itself, but from the understanding that prediction necessitates.

Tokens: The Building Blocks of Language

Before we can understand how language models process text, we need to examine how they represent language internally. Humans think in words, but language models work with tokens—subword units that form the atomic elements of their processing. Understanding tokens is crucial because virtually everything a language model does operates on tokens: input text is converted to tokens, calculations happen token by token, and output tokens are generated sequentially.

Modern language models typically don't use whole words as their basic units. There are several reasons for this. First, vocabulary size becomes problematic—if we tried to include every possible English word as a separate token, we'd need a vocabulary of hundreds of thousands of words, and we'd still struggle with misspellings, neologisms, and domain-specific terminology. Second, many words share meaningful subword components—think of "running," "runner," "runs," and "runnable." Treating these as completely separate tokens misses important patterns about their shared structure and meaning.

The most common approach to tokenization uses a technique called Byte Pair Encoding (BPE) or its variants. BPE starts with individual characters as tokens and repeatedly merges the most frequently occurring adjacent pairs into new tokens. This process continues until reaching a target vocabulary size, typically somewhere between 30,000 and 100,000 tokens.

Here's a simplified example of how BPE might work on a small corpus:

``` Initial tokens: [h, e, l, l, o, _, w, o, r, l, d] [i, _, a, m, _, _, _, _, _, _, _]

Most common pair: "l" + "l" → merge to "ll" After merge: [h, e, ll, o, _, w, o, r, ll, d] [i, _, a, m, _, _, _, _, _, _, _]

Most common pair: "o" + "_" → merge to "o_" After merge: [h, e, ll, o_, w, o_, r, ll, d] [i, _, a, m, _, _, _, _, _, _, _] ```

Through this process, common words like "hello" and "world" might become single tokens (or nearly so), while less common words get broken into smaller pieces. The word "tokenization" itself might be broken into tokens like ["token", "ization"], while common words like "the" or "a