Foundations of Generative Modeling: Probability, Divergence, and the Learning Objective

The quest to model and replicate the complex distributions of natural data—be it images, text, or audio—lies at the heart of modern artificial intelligence. Generative modeling addresses a fundamental question: given a finite set of observed samples from an unknown data distribution $p_{\text{data}}(x)$, how can we construct a model that captures its essential structure and variability, enabling us to generate new, plausible samples? The answer does not reside in a single algorithm but in a probabilistic framework that formalizes this objective as one of distribution matching. This chapter establishes that formal framework. We move beyond heuristic descriptions to the precise mathematical language that defines the generative modeling problem, distinguishes between major modeling paradigms, and specifies the optimization objectives that drive learning. The concepts of divergence, likelihood, and the implicit manifold hypothesis introduced here will serve as the foundational pillars upon which the architectures of GANs, diffusion models, and their predecessors are built and compared.

The Probabilistic Formulation of Generation

At its core, generative modeling is a problem of statistical inference. We assume the observed data $\mathcal{D} = \{x^{(1)}, x^{(2)}, \dots, x^{(N)}\}$ are independent and identically distributed (i.i.d.) samples drawn from an underlying, fixed but unknown probability distribution $p_{\text{data}}(x)$ defined over some data space $\mathcal{X}$. For images, $\mathcal{X}$ might be the space of all possible pixel arrays (e.g., $\mathbb{R}^{H \times W \times C}$ for continuous intensity values or a discrete set for quantized images). The generative model, denoted $p_{\text{model}}(x; \theta)$, is a parametric family of probability distributions indexed by parameters $\theta$. The learning process is the search for parameters $\theta^*$ such that $p_{\text{model}}(x; \theta^*)$ is "close" to $p_{\text{data}}(x)$ in a well-defined sense. This "closeness" is quantified by a divergence measure, a function $D(p || q)$ that assigns a non-negative scalar to two distributions, with $D(p || q) = 0$ if and only if $p = q$ almost everywhere.

This formulation immediately raises two critical questions: (1) How do we represent and compute with $p_{\text{model}}(x; \theta)$? (2) Which divergence $D$ should we minimize, and what are its statistical and computational properties? The answer to the first question bifurcates the field into two dominant paradigms: explicit density models and implicit density models.

Explicit Density Models: The Likelihood Path

Explicit density models directly parameterize and provide a tractable formula for the probability density (or mass) function $p_{\text{model}}(x; \theta)$. This allows for the direct computation of the likelihood of any data point under the model. The most common learning objective is the maximum likelihood estimation (MLE). Given i.i.d. data, the log-likelihood is:

$$\log p_{\text{model}}(\mathcal{D}; \theta) = \sum_{i=1}^{N} \log p_{\text{model}}(x^{(i)}; \theta)$$

MLE seeks $\theta_{\text{MLE}} = \arg\max_{\theta} \log p_{\text{model}}(\mathcal{D}; \theta)$. Under mild conditions, MLE is consistent (the learned distribution converges to $p_{\text{data}}$ as $N \to \infty$) and efficient (asymptotically achieves the lowest possible variance among unbiased estimators). However, its success is entirely contingent on the tractability of $p_{\text{model}}(x; \theta)$ and its gradient with respect to $\theta$.

The history of explicit models is a history of architectural ingenuity to maintain tractability while increasing expressive power. Autoregressive models (e.g., PixelCNN, WaveNet) factorize the joint distribution into a product of conditionals: $p(x) = \prod_{i=1}^{d} p(x_i | x_{<i})$, where the ordering defines a causal chain. This guarantees tractable likelihood and sampling (via sequential generation), but sampling is inherently sequential and thus slow. Flow-based models (e.g., NICE, RealNVP, Glow) construct $p_{\text{model}}$ through a sequence of invertible, differentiable transformations $f$ applied to a simple base distribution (e.g., standard Gaussian): $x = f(z)$, with $z \sim p_z(z)$. The change of variables formula provides an exact, tractable log-likelihood: $\log p(x) = \log p_z(z) + \log |\det(\partial f / \partial z)|$. These models support both efficient likelihood computation and efficient, parallel sampling via $z \mapsto f^{-1}(z)$. Their limitation is the architectural constraint of invertibility, which can limit expressiveness or require complex coupling layers. Variational Autoencoders (VAEs) represent a different approach: they posit an intractable true posterior $p(z|x)$ and introduce a tractable approximate posterior $q_{\phi}(z|x)$ (the encoder). The model learns by maximizing a variational lower bound (ELBO) on the log-likelihood: $\log p(x) \geq \mathbb{E}_{z \sim q_{\phi}(z|x)}[\log p_{\theta}(x|z)] - D_{\text{KL}}(q_{\phi}(z|x) || p(z))$. Here, $p_{\theta}(x|z)$ is the decoder (often a neural network), and $p(z)$ is a simple prior. The ELBO is tractable to optimize but introduces a bias: the true likelihood is lower-bounded, not computed exactly. VAEs trade exact likelihood for a more flexible, latent-variable structure that often yields better samples than early autoregressive models, though they historically suffered from blurriness.

The common thread for explicit models is the density estimation paradigm: we can ask the model "what is the probability of this exact sample?" This property is invaluable for tasks like anomaly detection (low likelihood indicates outlier) and principled model comparison (via held-out likelihood). However, the requirement of a tractable, normalized density severely restricts the model family. It is this restriction that motivated the search for alternatives, leading to the rise of implicit models.

Implicit Density Models: The Samples-Only Path

Implicit density models define a procedure for drawing samples from $p_{\text{model}}(x; \theta)$ without ever explicitly specifying a formula for its probability density function. The model's "definition" is the sampling process itself. The most prominent example is the family of Generative Adversarial Networks (GANs). A GAN consists of two networks: a generator $G_{\theta}: \mathcal{Z} \to \mathcal{X}$ that maps a noise variable $z \sim p_z(z)$ (e.g., $\mathcal{N}(0, I)$) to a data sample, and a discriminator $D_{\phi}: \mathcal{X} \to [0,1]$ that estimates the probability that a sample came from $p_{\text{data}}$ rather than $p_{\text{model}}$. The generator's parameters $\theta$ are trained to fool the discriminator, while the discriminator's parameters $\phi$ are trained to distinguish real from generated samples. This adversarial game, under ideal conditions and with sufficient capacity, converges to a generator that produces samples from $p_{\text{data}}$.

The key advantage of implicit models is expressive flexibility. The generator can be any differentiable, deterministic function of noise—a deep neural network with no architectural constraints like invertibility or autoregressive ordering. This allows for highly complex, multi-modal distributions to be approximated, potentially with simpler architectures than required for tractable explicit density estimation. The trade-off is profound: we lose the ability to compute $p_{\text{model}}(x; \theta)$ for any $x$. We cannot evaluate likelihoods, compute marginal probabilities, or directly apply likelihood-based model selection. The model is defined solely by its samples. This has practical consequences: GANs are notoriously difficult to train (mode collapse, instability), and their evaluation relies on heuristic sample quality metrics (Inception Score, Fréchet Inception Distance) rather than principled likelihoods.

The implicit model framework is broader than GANs. Any procedure that draw