Foundations of Generative Modeling: Probability, Divergence, and the Learning Objective

The fundamental challenge of generative modeling is to learn an underlying probability distribution from observed data such that new samples can be drawn that are indistinguishable from those observed. Formally, given a dataset $\mathcal{D} = \{x_1, x_2, \dots, x_N\}$ assumed to be drawn i.i.d. from an unknown data distribution $p_{\text{data}}(x)$, the goal is to construct a model $p_{\text{model}}(x; \theta)$ parameterized by $\theta$ that approximates $p_{\text{data}}(x)$ as closely as possible. This seemingly simple objective belies profound mathematical and computational complexities that have driven decades of research in machine learning and statistics. The ability to accurately model high-dimensional, structured distributions—such as those governing natural images, speech, or molecular structures—has become a cornerstone of modern artificial intelligence, enabling applications ranging from creative content synthesis to scientific discovery. Yet, the path to effective generative models is fraught with trade-offs: between tractability and expressiveness, between computational efficiency and statistical accuracy, and between theoretical guarantees and practical performance. This chapter establishes the core probabilistic and mathematical framework that underpins all generative models, regardless of their specific architectural instantiation. We will rigorously define the learning objective as a problem of distribution matching, introduce the key divergence measures used to quantify the difference between distributions—notably Kullback-Leibler (KL) divergence, Jensen-Shannon (JS) divergence, and Wasserstein distance—and elucidate the critical distinction between explicit and implicit generative models. This foundation is essential for understanding why generative adversarial networks (GANs) and diffusion models represent two fundamentally different, yet ultimately convergent, approaches to solving the same core problem: how to learn and sample from complex data distributions in high-dimensional spaces.

1.1 Probability Theory: The Language of Uncertainty

Before delving into generative models, we must ground ourselves in the mathematical language of probability, which provides the formal mechanism for describing uncertainty and stochastic processes. At its heart, a probability space $(\Omega, \mathcal{F}, P)$ consists of a sample space $\Omega$ (all possible outcomes), a $\sigma$-algebra $\mathcal{F}$ (events), and a probability measure $P$ assigning likelihoods to events. For continuous random variables, which are the domain of most modern generative models (e.g., pixel values in an image, amplitude samples in audio), we work with probability density functions (PDFs). A random variable $X$ is a measurable function from $\Omega$ to $\mathbb{R}^d$; its distribution is described by a density $p(x)$ such that $P(X \in A) = \int_A p(x) dx$ for any measurable set $A \subseteq \mathbb{R}^d$. The density must satisfy $p(x) \geq 0$ and $\int_{\mathbb{R}^d} p(x) dx = 1$. Joint distributions over multiple variables factorize according to conditional probabilities via the chain rule: $p(x_1, \dots, x_d) = p(x_1) p(x_2|x_1) \cdots p(x_d|x_1,\dots,x_{d-1})$. This factorization is exploited by autoregressive models, as we shall see.

In generative modeling, we assume our dataset $\mathcal{D}$ consists of $N$ independent and identically distributed (i.i.d.) samples from $p_{\text{data}}(x)$. The i.i.d. assumption simplifies theoretical analysis and is approximately true for many real-world datasets after appropriate preprocessing (e.g., shuffling). The central task is then to estimate $p_{\text{data}}$ from finite samples. This is an instance of density estimation, a well-studied problem in statistics. Classical approaches include parametric methods (e.g., fitting a Gaussian distribution by maximum likelihood) and non-parametric methods (e.g., kernel density estimation). However, these traditional techniques struggle in high dimensions due to the curse of dimensionality: the number of samples required to estimate a density grows exponentially with dimension. For instance, to achieve a fixed estimation error in $d$ dimensions with a kernel density estimator, the required sample size scales as $N = \mathcal{O}(h^{-d})$, where $h$ is the bandwidth; even for moderate $d$ (e.g., $d=100$ for a small image), this becomes infeasible. Modern deep generative models circumvent this curse by leveraging neural networks to impose inductive biases—such as hierarchical feature representations and compositional structure—that enable efficient learning from comparatively fewer samples. Nevertheless, the fundamental goal remains unchanged: to approximate the true data distribution as closely as possible, either by directly modeling its density or by generating samples that follow it.

1.2 Explicit vs. Implicit Generative Models

Generative models can be broadly categorized into two paradigms based on how they represent and interact with the target distribution: explicit models and implicit models. This distinction is crucial because it determines the mathematical tools available for training, evaluation, and inference.

Explicit models provide a direct, tractable specification of the probability density function $p_{\text{model}}(x; \theta)$. Given a parameter vector $\theta$, one can evaluate $p_{\text{model}}(x; \theta)$ for any input $x \in \mathbb{R}^d$. This explicit density enables exact computation of likelihoods, which can be maximized directly via maximum likelihood estimation (MLE): $\theta^* = \arg\max_\theta \frac{1}{N} \sum_{i=1}^N \log p_{\text{model}}(x_i; \theta)$. Explicit models include:

• **Fully visible belief networks (autoregressive models)**: These decompose the joint density via the chain rule: $p(x) = \prod_{i=1}^d p(x_i | x_{<i})$. Each conditional is modeled by a neural network (e.g., PixelCNN for images, WaveNet for audio). They allow exact likelihood computation but suffer from sequential generation, making sampling slow.
• **Variational autoencoders (VAEs)**: These introduce latent variables $z$ and define a generative process $p_\theta(x) = \int p_\theta(x|z) p(z) dz$. The integral is intractable, but a variational approximation $q_\phi(z|x)$ yields a lower bound on the log-likelihood (the ELBO). VAEs provide a tractable objective but often produce blurry samples due to the approximation and the tendency of the KL term to force the approximate posterior toward the prior.
• **Normalizing flows**: These use a sequence of invertible, differentiable transformations $f_\theta$ to map a simple base density $p_z(z)$ (e.g., standard Gaussian) to the data density: $p_\theta(x) = p_z(f_\theta^{-1}(x)) \left| \det \frac{\partial f_\theta^{-1}(x)}{\partial x} \right|$. The change-of-variables formula gives an exact density, and sampling is fast (just apply $f_\theta$ to $z$). However, designing expressive yet efficient invertible architectures is challenging.

The advantage of explicit models is the ability to compute exact likelihoods, which facilitates rigorous statistical evaluation, principled model comparison (via held-out log-likelihood), and straightforward integration into larger probabilistic systems. Their drawback is often the need for restrictive architectural choices (e.g., autoregressive factorization limits parallel computation) or approximations (e.g., VAEs) that can degrade sample quality.

Implicit models, in contrast, do not provide an explicit formula for $p_{\text{model}}(x; \theta)$. Instead, they define a stochastic procedure that generates samples: draw a random noise vector $z \sim p_z(z)$ (typically a simple distribution like uniform or Gaussian), then apply a deterministic transformation $G_\theta(z)$ (the generator) to produce a sample $x = G_\theta(z)$. The induced distribution $p_G(x) = \mathbb{E}_{z \sim p_z}[ \delta(x - G_\theta(z)) ]$ is implicitly defined by the generator's mapping. No density evaluation is poss