Foundations of Generative Modeling: Likelihood, Divergence, and Assessment

Generative modeling stands as one of the most transformative pillars of modern artificial intelligence, enabling machines to synthesize realistic data—images, text, audio, and beyond—by learning the underlying probability distribution of observed examples. At its core, the generative task is deceptively simple: given a dataset $\mathcal{D} = \{x^{(i)}\}_{i=1}^N$ drawn from an unknown true distribution $p_{\text{data}}(x)$, construct a model $p_\theta(x)$ that approximates $p_{\text{data}}(x)$ so well that new samples $x \sim p_\theta(x)$ are indistinguishable from real data. Yet this simplicity belies profound theoretical and practical challenges. How do we measure the distance between two high-dimensional distributions? How can we optimize intractable objectives? And once a model is trained, how do we evaluate whether its outputs are *good*—a question that quickly spirals into philosophy when “good” must be defined without a single ground-truth answer? This chapter establishes the rigorous probabilistic and statistical foundations that underpin every generative architecture from early autoregressive models to today’s diffusion frontiers. We will dissect the primary families of training objectives—exact likelihood, variational inference, score matching, and integral probability metrics—deriving their mathematical relationships and exposing the architectural trade-offs they impose. Crucially, we will then turn to the often-overlooked but essential problem of assessment: a critical analysis of standard metrics like Inception Score (IS), Fréchet Inception Distance (FID), Kernel Inception Distance (KID), and CLIP Score, revealing their hidden assumptions, failure modes, and the subtle ways they can mislead research directions. By the end of this chapter, you will not only possess a toolkit of formal methods but also a seasoned skepticism toward evaluation practices—a dual perspective necessary for advancing the field responsibly.

---

1.1 The Probabilistic Foundation: Modeling Data as Distributions

Before diving into algorithms, we must cement the conceptual framework: data generation is a stochastic process. Every image of a handwritten digit, every sentence in a corpus, every frame of a video is a sample from some complex, structured distribution $p_{\text{data}}(x)$ defined over a space $\mathcal{X}$ (e.g., $\mathbb{R}^{H \times W \times 3}$ for images). The generative model’s goal is to learn a parametric family $p_\theta(x)$, where $\theta \in \Theta$ are tunable parameters, such that $p_\theta \approx p_{\text{data}}$. This approximation can be evaluated using a divergence or distance measure $D(p_\theta \| p_{\text{data}})$, but $p_{\text{data}}$ is unknown—we only have samples. Thus, all learning algorithms reduce to optimizing an objective $\mathcal{L}(\theta)$ that is either a proxy for $D$ or directly tractable from data.

The choice of objective fundamentally shapes the model’s architecture, training dynamics, and sample quality. We can broadly categorize objectives by how they relate to the *density* (explicit $p_\theta(x)$) or *score* ($\nabla_x \log p_\theta(x)$) of the model:

• **Density-based methods** directly model $p_\theta(x)$ and optimize its likelihood on data.
• **Score-based methods** model the gradient of the log-density (the score function) and minimize a divergence equivalent to matching scores.
• **Divergence-based methods** (e.g., GANs) forgo explicit densities and minimize a distance between $p_\theta$ and $p_{\text{data}}$ without requiring density estimation.

Each path carries distinct computational burdens and expressive constraints, which we now explore in depth.

---

1.2 Exact Likelihood: The Gold Standard and Its Limits

The most statistically efficient estimator for a parametric model is the maximum likelihood estimator (MLE), which maximizes the log-likelihood of observed data under $p_\theta$:

$$ \theta^* = \arg\max_\theta \sum_{i=1}^N \log p_\theta(x^{(i)}). $$

If $p_\theta$ is correctly specified, MLE is consistent, asymptotically normal, and efficient. However, exact likelihood requires that $p_\theta(x)$ is *tractable*: we must be able to compute $\log p_\theta(x)$ and $\nabla_\theta \log p_\theta(x)$ for any $x$. This is feasible only for models with a constrained structure, typically those that factorize the joint distribution into a product of conditional distributions via a *fixed* ordering of variables—the autoregressive approach.

1.2.1 Autoregressive Models: A Chain of Conditional Distributions

For a data point $x = (x_1, x_2, \dots, x_d)$ (e.g., pixels flattened into a vector, or tokens in a sequence), an autoregressive model decomposes the joint as:

$$ p_\theta(x) = \prod_{k=1}^d p_\theta(x_k \mid x_{<k}), $$

where $x_{<k} = (x_1, \dots, x_{k-1})$. Each conditional $p_\theta(x_k \mid x_{<k})$ is typically modeled by a neural network (e.g., an RNN, Transformer, or CNN) that takes $x_{<k}$ as input and outputs parameters of a simple distribution (e.g., discrete softmax for pixels, or Gaussian for continuous data). The total log-likelihood becomes a sum of per-position log-likelihoods, all computable in a single forward pass.

Example: PixelCNN — A seminal autoregressive image model uses a masked convolutional network to ensure that the prediction for pixel $x_k$ depends only on pixels above and to the left (a fixed raster order). The network stacks layers of masked convolutions, and each output channel parameterizes a discrete distribution (e.g., a 256-way softmax for 8-bit images). Training is straightforward: minimize the negative log-likelihood (NLL) via stochastic gradient descent. Sampling, however, is sequential and slow: we must generate pixels one by one in the predefined order.

The computational cost of autoregressive sampling—$O(d)$ steps per sample—makes it impractical for high-resolution images or long sequences, despite excellent density estimation performance (often achieving the lowest bits-per-dimension on benchmarks). This reveals the first major trade-off: exact likelihood models excel at density estimation but suffer from slow, ordered generation.

1.2.2 Normalizing Flows: Exact Likelihood with Parallel Generation

Normalizing flows offer a compelling alternative: they transform a simple base density $p_z(z)$ (e.g., standard Gaussian) through a sequence of invertible, differentiable functions $f_1, f_2, \dots, f_K$ with parameters $\theta$. The final transformation $x = f_K \circ \cdots \circ f_1(z)$ yields a complex density $p_\theta(x)$ given by the change-of-variables formula:

$$ p_\theta(x) = p_z(f^{-1}(x)) \prod_{k=1}^K \left| \det \frac{\partial f_k^{-1}}{\partial x} \right|, $$

where $f = f_K \circ \cdots \circ f_1$ and $\det$ denotes the Jacobian determinant. Because the transformation is invertible, we can compute $\log p_\theta(x)$ exactly in a single pass—no sequential sampling required. Sampling is also parallel: draw $z \sim p_z(z)$, then compute $x = f(z)$.

The challenge lies in designing transformations that are both *expressive* (able to model complex data distributions) and *computationally efficient* ( Jacobian determinants must be cheap to compute). Early flows like RealNVP used coupling layers that split dimensions and apply affine transformations conditioned on the other split, yielding a Jacobian that is triangular with ones on the diagonal (determinant = 1). Later advances like Glow introduced invertible $1 \times 1$ convolutions and actnorm layers to boost performance. More recent continuous normalizing flows (CNFs) model the transformation as a solution to an ordinary differential equation (ODE), allowing infinite-depth flows with theoretically universal approximation capabilities.

Despite their elegance, flows have struggled to match the sample quality of GANs or diffusion models on high-resolution image synthesis. The constraint of invertibility limits the model’s ability to discard irrelevant inf