Foundations of Generative Modeling: Likelihood, Divergence, and the Learning Objective

Generative modeling represents one of the most profound shifts in how machines understand and create data. Unlike discriminative models that map inputs to labels, generative models seek to capture the underlying probability distribution of the data itself—the very fabric from which observations are woven. This chapter establishes the rigorous probabilistic foundation upon which all modern generative AI is built, from the earliest autoregressive models to today's multimodal diffusion systems. We will begin with the fundamental question: what does it mean for a machine to "learn" a distribution? The answer lies in the elegant interplay between likelihood, divergence, and optimization—a triad that forms the conceptual backbone of this entire field. Understanding these principles is not merely academic; it is a practical necessity for anyone seeking to design, train, or debug generative models at scale. The choices we make in defining a learning objective—whether to maximize likelihood, minimize a divergence, or engage in adversarial games—directly shape what a model can and cannot do, its sample quality, its mode coverage, and its very notion of "reality." As we journey from the crisp mathematics of explicit density estimation to the nuanced博弈 of implicit models, we will see how each approach embodies a different philosophy about what it means to approximate the true data distribution $p_{data}(x)$. This chapter provides the essential toolkit for navigating that landscape: the definitions, theorems, and intuitions that will recur throughout this book, whether we are analyzing a simple variational autoencoder or a state-of-the-art multimodal diffusion transformer.

Probability Fundamentals: The Language of Generation

At its core, generative modeling is applied probability theory. Before we can discuss how to learn a distribution, we must be precise about what a distribution is and how we describe it mathematically. Let $\mathcal{X}$ denote the data space—this could be the space of all possible $28 \times 28$ grayscale images, the space of English sentences, or the space of audio waveforms. A probability distribution $p(x)$ over $\mathcal{X}$ assigns a likelihood to every possible data point $x \in \mathcal{X}$. For continuous spaces, we work with probability density functions (PDFs) $p(x)$ such that $\int_{\mathcal{X}} p(x) dx = 1$; for discrete spaces, we use probability mass functions (PMFs) $p(x)$ with $\sum_{x \in \mathcal{X}} p(x) = 1$. The true data distribution $p_{data}(x)$ is the unknown distribution from which our finite training dataset $\{x^{(i)}\}_{i=1}^N$ is assumed to be independently and identically distributed (i.i.d.). The generative modeling problem is: given only samples from $p_{data}$, construct a model $p_\theta(x)$ (where $\theta$ denotes parameters) that approximates $p_{data}$ as closely as possible, according to some metric.

Two primary schools of thought emerge from this definition. Explicit density models aim to directly specify a parametric functional form for $p_\theta(x)$ that is *normalized*—meaning $\int p_\theta(x) dx = 1$ (or sums to 1) and for which we can evaluate the density $p_\theta(x)$ for any input $x$. Examples include Gaussian mixture models, autoregressive models like PixelCNN, and normalizing flows. The key advantage is that we can compute the exact likelihood of any data point, which enables principled probabilistic reasoning, outlier detection, and a clear optimization objective: maximum likelihood estimation (MLE). The likelihood of the observed data under the model is $p_\theta(\mathcal{D}) = \prod_{i=1}^N p_\theta(x^{(i)})$, and MLE seeks $\theta^* = \arg\max_\theta \sum_{i=1}^N \log p_\theta(x^{(i)})$. This is equivalent to minimizing the expected negative log-likelihood, or the cross-entropy between the empirical data distribution $\hat{p}_{data}$ and the model $p_\theta$.

Implicit density models, by contrast, do not provide a closed-form expression for $p_\theta(x)$. Instead, they define a stochastic *generative process* that maps samples $z$ from a simple, fixed prior distribution $p(z)$ (e.g., standard Gaussian) to samples $x = g_\theta(z)$ in the data space. The model distribution $p_\theta(x)$ is then defined *implicitly* as the distribution induced by this mapping: $p_\theta(x) = \int \delta(x - g_\theta(z)) p(z) dz$, where $\delta$ is the Dirac delta function. Since we can sample from $p_\theta$ by first sampling $z \sim p(z)$ and then computing $x = g_\theta(z)$, but we generally cannot evaluate $p_\theta(x)$ for a given $x$ (except perhaps through expensive approximation), these models are not directly amenable to likelihood-based training. Generative Adversarial Networks (GANs) and many diffusion models fall into this category. Their training objectives must therefore be derived from other principles, often involving the comparison of real and generated samples without ever needing to compute explicit densities.

This explicit/implicit dichotomy is fundamental. It influences everything from model architecture to evaluation metrics. Explicit models often come with strong theoretical guarantees but may struggle with flexibility or computational cost (e.g., the sequential nature of autoregressive models). Implicit models offer incredible flexibility and often produce sharper samples but can be harder to train and evaluate, lacking a direct density estimate. Later chapters will explore these trade-offs in depth, but for now, we must establish the mathematical tools to compare any two distributions—explicit or implicit—which brings us to the concept of divergence.

Divergence Measures: Quantifying Distributional Distance

How do we measure how "close" two probability distributions $P$ and $Q$ are? The answer is not unique; different divergence measures capture different notions of similarity and lead to different learning dynamics. A divergence $D(P||Q)$ is a function that quantifies the difference between two distributions, typically satisfying $D(P||Q) \geq 0$ with equality if and only if $P = Q$ almost everywhere. However, divergences are not necessarily symmetric ($D(P||Q) \neq D(Q||P)$) and do not necessarily obey the triangle inequality. The choice of divergence is a *modeling decision* that encodes our preferences: do we care more about covering all modes of the true distribution, or about producing high-quality samples in the modes we do cover? Different divergences prioritize these goals differently.

Kullback-Leibler Divergence: The Asymmetric Workhorse

The Kullback-Leibler (KL) divergence is the most ubiquitous divergence in machine learning. For two distributions $P$ and $Q$ with densities $p(x)$ and $q(x)$, it is defined as:

$$D_{KL}(P || Q) = \mathbb{E}_{x \sim P} \left[ \log \frac{p(x)}{q(x)} \right] = \int p(x) \log \frac{p(x)}{q(x)} dx$$

This can be rewritten as $D_{KL}(P || Q) = \mathbb{E}_{x \sim P}[\log p(x)] - \mathbb{E}_{x \sim P}[\log q(x)]$. The first term is the entropy of $P$, $H(P) = -\mathbb{E}_{P}[\log p(x)]$, and the second term is the cross-entropy $H(P, Q) = -\mathbb{E}_{P}[\log q(x)]$. Thus $D_{KL}(P || Q) = H(P, Q) - H(P)$. The KL divergence measures the expected extra information required to encode samples from $P$ when using a code optimized for $Q$. It is always non-negative by Jensen's inequality, and it is zero iff $P = Q$ almost surely.

The critical property of KL divergence is its asymmetry. $D_{KL}(P || Q)$ and $D_{KL}(Q || P)$ capture fundamentally different behaviors. Consider a simple example: let $P$ be a bimodal distribution with two well-separated modes, and let $Q$ be a unimodal distribution centered under one of the modes. Minimizing $D_{KL}(P || Q)$ forces $Q$ to cover *all* the mass that $P$ places anywhere, because the expectation is taken under $P$. If $Q$ assigns low probability to a region where $P$ has mass, the term $\log \frac{p(x)}{q(x)}$ becomes very large and positive, heavily penalizing that mism