What Is a Language Model? The Foundation of Thinking Machines

Have you ever played the game where someone whispers a sentence to the person next to them, and that person whispers it to the next, and so on? By the time the sentence reaches the last person, it often sounds completely different from what it started as. This children's game actually teaches us something profound about language: small changes in how we hear or interpret words can completely transform their meaning. Now imagine trying to teach a machine not just to repeat sentences, but to understand them—to know that "The cat sat on the mat" makes sense while "The cat sat on the cloud" feels slightly strange, even though both sentences follow the basic rules of grammar. This is exactly the challenge that language models attempt to solve, and understanding how they work will take us on one of the most fascinating adventures in modern technology.

Before we can appreciate the remarkable abilities of today's Large Language Models, we need to understand what a language model actually is at its most fundamental level. At its core, a language model is a mathematical system that has learned, through exposure to vast amounts of text, to assign probabilities to sequences of words. In simpler terms, when given a sequence of words that someone might want to say or write, a language model can estimate how likely it is that this particular sequence would actually appear in real language. This probability-based approach to understanding language might seem abstract at first, but it becomes much clearer when we consider a familiar example from everyday life: your smartphone's keyboard.

Think about the last time you typed a message on your phone. Did you notice how it sometimes suggests the next word before you even finish typing? When you type "I'm feeling very," your phone might suggest "happy," "tired," "sick," or "good" as possible next words. These suggestions appear because your phone contains a small, simple language model that has learned which words tend to follow which other words. The suggestion "happy" might appear more frequently not because your phone knows what emotions are, but because the phrase "feeling very happy" appears millions of times in all the text data that was used to train it. The word "happy" has a higher probability of following the phrase "feeling very" compared to other words, and your phone's keyboard is essentially making educated guesses based on these learned probabilities.

This probabilistic approach to language might seem overly mechanical, almost like a sophisticated guessing game rather than true understanding. Yet researchers have discovered that by learning statistical patterns in language—patterns about which words tend to appear together, which phrases sound natural, and which combinations of ideas commonly occur—machines can develop surprisingly sophisticated linguistic capabilities. A language model that has seen enough examples of the phrase "The sun rises in the" will learn that "east" is far more likely to follow than "pizza" or "happiness." It has never seen the sun rise in the pizza, but it knows that humans almost always write about the sun rising in the east. This statistical knowledge, accumulated across billions and trillions of sentences, forms the foundation upon which all language models are built.

The distinction between a simple language model and a Large Language Model lies not in their fundamental approach—both operate on the principle of predicting what comes next based on learned patterns—but rather in their scale, sophistication, and the remarkable capabilities that emerge from that scale. A simple language model, like the one in your phone's keyboard, might be trained on relatively modest amounts of text and can only look at a few words at a time when making predictions. It knows that "hungry" often follows "I'm so" and that "please" often precedes "thank you," but it struggles with more complex linguistic relationships that require understanding longer stretches of text. A Large Language Model, on the other hand, represents a fundamentally different tier of capability, trained on internet-scale collections of text and containing billions of numerical parameters that encode its understanding of language.

When we speak of a language model being "large," we are referring primarily to two things: the size of its training data and the number of its parameters. Training data refers to all the text that the model learns from during its development. A simple language model might be trained on a collection of books, perhaps a few hundred million words in total. A Large Language Model, by contrast, might be trained on trillions of words—essentially the entire accessible written output of human civilization, including books, articles, websites, scientific papers, software code, and countless other forms of text. This astronomical quantity of training data exposes the model to an extraordinarily diverse range of language patterns, topics, writing styles, and ways of expressing ideas.

The parameters of a language model can be understood as the learned "knowledge" that the model has extracted from its training data. If we use an analogy from human biology, we might compare parameters to the synaptic connections in our brains. When you learn to ride a bicycle, your brain doesn't store a set of instructions like a manual; instead, it creates and strengthens certain neural connections while weakening others. Over time and with practice, these connections encode your learned ability to balance, pedal, and steer. Similarly, when a Large Language Model is trained, it doesn't store sentences or rules about language in any way that resembles a dictionary or grammar book. Instead, it adjusts billions of numerical values—its parameters—in ways that allow it to predict language patterns. Each parameter is a number, and the intricate web of relationships between these numbers encodes everything the model has learned about how language works.

To make this more concrete, consider the magnitude of numbers involved. Early language models from the 1990s and early 2000s might have had thousands or hundreds of thousands of parameters. By 2018, models with hundreds of millions of parameters were considered large. Today, state-of-the-art Large Language Models contain hundreds of billions of parameters. GPT-3, one of the most well-known LLMs, contains approximately 175 billion parameters. These aren't arbitrary increases in size for their own sake; each additional parameter provides additional capacity for the model to learn and represent nuanced patterns in language. The relationship between parameter count and capability is not perfectly linear—there are diminishing returns and surprising threshold effects—but larger models with more parameters consistently demonstrate abilities that smaller models cannot achieve.

Understanding parameters requires us to venture into the realm of neural networks, the computational architecture that underlies all modern language models. A neural network, despite its name evoking biological neurons, is actually a mathematical structure composed of layers of interconnected processing units. Each connection between units has a weight—a parameter—that determines how much influence one unit's output has on another's. When we say a model has billions of parameters, we mean that it has billions of these connection weights, all of which are adjusted during training to minimize errors in the model's predictions. The process of training involves showing the model enormous quantities of text, asking it to predict what comes next, measuring how wrong its predictions are, and then adjusting all its parameters slightly to make better predictions in the future. This process is repeated countless times until the model's predictions become remarkably accurate.

The neural network architecture that powers contemporary LLMs is called the Transformer. Introduced in a landmark 2017 research paper, the Transforme