What Are Large Language Models? A Gentle Introduction

Understanding Large Language Models: A Gentle Introduction

Imagine sitting in a cozy café, listening to two people have a conversation. One person says, "I think we should—" and the other immediately responds, "—postpone the launch until next quarter?" The second person didn't wait for the first to finish because language is remarkably predictable. We constantly anticipate what others will say next based on context, experience, and patterns we've absorbed throughout our lives. This natural ability to predict and generate language is exactly what large language models (LLMs) attempt to replicate through computation.

When OpenAI released ChatGPT in November 2022, it sparked a conversation that spread from research laboratories to kitchen tables worldwide. Within five days, the service attracted one million users—a growth rate that took Netflix three and a half years to achieve. By early 2024, ChatGPT had become one of the fastest services in history to reach 100 million monthly active users. Similar AI assistants from Anthropic (Claude), Google (Gemini, formerly Bard), Meta (Llama), and numerous other companies followed, each pushing the boundaries of what machines could accomplish with human language.

This book explores the technology behind this revolution: the transformer architecture that powers these systems, the principles that guide their development, and the practical implications of machines that can engage in seemingly intelligent conversation. But before diving into the technical details, we need to establish a clear foundation. What exactly are large language models? How do they work at a conceptual level? And what should you realistically expect when interacting with them?

This chapter provides that foundation. We'll build your understanding step by step, starting from simple analogies and progressively introducing the key concepts and terminology you'll need throughout this book. By the end, you'll possess a solid mental model of what LLMs are and how they function—not as mystical artificial general intelligence, but as sophisticated pattern-matching systems trained on vast quantities of text.

The Simple Idea Behind Language Models

At its core, a language model is a system that predicts what comes next in a sequence of text. Consider the sentence: "The sun rises in the ____." Almost certainly, your brain completed that blank with "east" or something equivalent. You did this effortlessly, drawing on your understanding of how language works and what makes sense in context.

Language models perform this same basic task computationally. Given a sequence of words or characters, they calculate probabilities for what might come next. A simple language model might recognize that after "sun rises in the," the word "east" is statistically very likely. A more sophisticated model considers broader context—"The scientist observed that the sun rises in the ____"—and might complete it with "east" as before, or perhaps with "polar region" if the preceding context established an Arctic setting.

This prediction task is called language modeling, and it forms the fundamental objective that LLMs are trained to accomplish. The word "model" here carries its statistical meaning: a representation that captures patterns in data and allows us make predictions. A language model is thus a computational representation of patterns in language.

But why is this prediction task so valuable? Because to predict language effectively, a system must implicitly learn enormous amounts about how language works: grammar and syntax, semantics and meaning, reasoning patterns, factual knowledge, and even subtle aspects like tone and style. A system that can predict what comes next in diverse contexts necessarily has absorbed deep understanding of language structure and usage.

The "large" in large language model refers to the scale at which these systems operate. Modern LLMs contain billions of parameters—adjustable values that influence how the model makes predictions. They are trained on trillions of words of text extracted from books, articles, websites, and other sources. This scale allows them to capture the full breadth and subtlety of human language expression.

Building Blocks: Understanding Tokens, Embeddings, and Parameters

To understand how LLMs process text, we need to introduce three fundamental concepts: tokens, embeddings, and parameters. Each serves a crucial role in transforming human language into a format that computers can process and in enabling the model to make intelligent predictions.

Tokens: The atoms of language

When you read the word "learning," your eyes perceive six letters arranged in sequence. When an LLM processes this word, it first breaks it into smaller pieces called tokens. Tokens are the basic units that LLMs work with—not quite individual characters, and not quite whole words.

Most modern LLMs use a scheme called Byte Pair Encoding (BPE) or similar subword tokenization algorithms. These approaches break text into pieces based on statistical patterns in training data, balancing between representing common words as single tokens and breaking rare or complex words into smaller subword units.

Here's how this works in practice:

• Common words like "the," "is," and "cat" might each be represented as single tokens
• Less common words might be broken into subword pieces: "learning" could become "learn" + "ing," or perhaps "learn" + "ing" depending on the specific tokenizer
• Rare words might be decomposed into individual characters or small character sequences

This approach offers important advantages. It allows models to handle any text, even words they haven't seen before, by composing them from known subword units. It also keeps the vocabulary manageable—rather than having a separate token for every possible word (including misspellings and variations), the model works with a few tens of thousands of basic units that can be combined flexibly.

When you interact with an LLM, you might hear references to "token limits" or "context windows." These refer to how many tokens the model can process in a single interaction. Early LLMs might handle a few thousand tokens (roughly equivalent to a few thousand words of text). More recent systems can process hundreds of thousands of tokens, enabling them to work with entire books or lengthy document collections in a single conversation.

The concept of tokens is fundamental because it defines what the model "sees" when processing text. Each token becomes an input to the model's computational machinery, and the model's predictions are always about which token comes next.

Embeddings: Turning words into numbers

Once text is divided into tokens, the model must convert these tokens into a format suitable for mathematical computation. This is where embeddings come in.

An embedding is a list of numbers—a vector of typically hundreds or thousands of dimensions—that represents a token. Each token in the model's vocabulary gets its own unique embedding, learned during training. These embeddings capture semantic relationships: tokens with similar meanings or that appear in similar contexts will have embeddings that are mathematically close to each other.

Consider a simple two-dimensional example for illustration. We might position tokens on a plane where one axis represents "concreteness" (from abstract to concrete) and another represents "emotional valence" (from negative to positive). The word "cat" might score high on concreteness and neutral on emotional valence, while "democracy" might score low on concreteness (abstract) and moderately positive. Real embeddings operate in hundreds of dimensions, capturing far more nuanced relationships.

The magic of embeddings lies in how they enable mathematical operations that correspond to semantic relationships. In a well-trained embedding space, you can perform arithmetic like: "king" minus "man" plus "woman" approximately equal