Foundations of Generative Modeling

Foundations of Generative Modeling

The ability to synthesize novel, realistic data from learned representations stands as one of the most profound achievements in modern machine learning. Consider, for a moment, the implications: a system that has observed thousands of photographs can learn the underlying structure of visual reality sufficiently well to produce images that humans cannot distinguish from authentic photographs. This capability—generative modeling—represents far more than a technical curiosity; it embodies a fundamental shift in how machines can interact with and understand the world. Before we can appreciate the sophisticated architectures that enable this capability, from Generative Adversarial Networks (GANs) to Diffusion Models, we must first establish the mathematical and conceptual foundations upon which these systems are built. This chapter undertakes that essential work, developing the probabilistic framework, the understanding of latent spaces, and the historical context necessary to comprehend the technical journey ahead.

The Nature of Data and Probability Distributions

At the heart of generative modeling lies a deceptively simple question: what is the structure of the world we wish to generate? The data we observe—images, text, audio, video—does not appear randomly without rhyme or reason. Photographs of faces, for instance, exhibit intricate patterns: eyes tend to be positioned symmetrically, skin tones vary within certain ranges, and backgrounds often contain recognizable objects. These regularities constitute what we call the data distribution, formally denoted as $p_{\text{data}}(\mathbf{x})$, where $\mathbf{x}$ represents a single data point, such as the vectorized pixel values of an image.

Understanding this notation requires careful unpacking. In the context of image generation, $\mathbf{x}$ typically belongs to a high-dimensional space $\mathbb{R}^n$, where $n$ equals the number of pixels multiplied by the number of color channels. For a modest $256 \times 256$ RGB image, this space has dimension $196,608$. The data distribution $p_{\text{data}}(\mathbf{x})$ assigns probability densities across this space such that regions corresponding to realistic images have high density, while regions corresponding to nonsensical random pixel configurations have negligible density. Crucially, we do not have direct access to this true data distribution; we can only observe samples drawn from it.

This observation leads us to the central problem of generative modeling: we have a finite dataset $\mathcal{D} = \{\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_m\}$ containing $m$ samples independently drawn from the unknown distribution $p_{\text{data}}$. From these samples, we wish to learn a model $p_\theta(\mathbf{x})$ parameterized by $\theta$ that approximates the true data distribution. Once we have such a model, we can sample from it to produce novel data points that resemble the training data.

Random Variables and Their Properties

The mathematical machinery for reasoning about distributions begins with the concept of a random variable. A random variable $X$ is a function that maps outcomes from a sample space to real numbers (or vectors). When $X$ can take any value in a continuous range, we work with probability density functions (PDFs). For a continuous random variable $\mathbf{x} \in \mathbb{R}^n$, the density function $p(\mathbf{x})$ must satisfy two fundamental properties:

1. Non-negativity: $p(\mathbf{x}) \geq 0$ for all $\mathbf{x}$ 2. Normalization: $\int_{\mathbb{R}^n} p(\mathbf{x}) \, d\mathbf{x} = 1$

These properties ensure that the density function represents a valid probability measure. When we speak of $p_\theta(\mathbf{x})$, we are referring to a parametric family of densities indexed by the parameters $\theta$.

Consider a simple illustrative case: the multivariate Gaussian distribution, which serves as both a foundational model and a building block for more sophisticated approaches. The PDF of a $d$-dimensional Gaussian with mean vector $\boldsymbol{\mu} \in \mathbb{R}^d$ and covariance matrix $\boldsymbol{\Sigma} \in \mathbb{R}^{d \times d}$ is given by:

$$p(\mathbf{x}; \boldsymbol{\mu}, \boldsymbol{\Sigma}) = \frac{1}{(2\pi)^{d/2} |\boldsymbol{\Sigma}|^{1/2}} \exp\left(-\frac{1}{2}(\mathbf{x} - \boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x} - \boldsymbol{\mu})\right)$$

This distribution has several important properties that extend to more complex models. The mean vector $\boldsymbol{\mu}$ captures the central tendency of the distribution, while the covariance matrix $\boldsymbol{\Sigma}$ captures the pairwise relationships between dimensions. For a diagonal covariance matrix, the dimensions are independent; for a full covariance matrix, we can model correlations.

```python import numpy as np

def gaussian_pdf(x, mu, sigma): """ Compute the PDF of a multivariate Gaussian at point x.

Parameters: ----------- x : np.ndarray Point at which to evaluate the density, shape (d,) mu : np.ndarray Mean vector, shape (d,) sigma : np.ndarray Covariance matrix, shape (d, d)

Returns: -------- float : Probability density at x """ d = len(x) diff = x - mu

Compute the exponent term: -0.5 * (x-mu)^T * Sigma^{-1} * (x-mu)

precision = np.linalg.inv(sigma) exponent = -0.5 * diff @ precision @ diff

Compute the normalization constant

det_sigma = np.linalg.det(sigma) normalization = 1.0 / ((2 * np.pi) (d / 2) * det_sigma 0.5)

return normalization * np.exp(exponent)

Example usage with 2D Gaussian

mean = np.array([0.0, 0.0]) cov = np.array([[1.0, 0.5], [0.5, 1.0]]) # Correlated variables

x = np.array([0.5, 0.5]) density = gaussian_pdf(x, mean, cov) ```

Joint and conditional distributions provide the mathematical framework for understanding relationships within data. For two random variables $\mathbf{x}$ and $\mathbf{y}$, the joint distribution $p(\mathbf{x}, \mathbf{y})$ describes the probability density over both variables simultaneously. From this joint distribution, we can derive the marginal distribution of $\mathbf{x}$ by integrating (or summing, for discrete variables) over $\mathbf{y}$:

$$p(\mathbf{x}) = \int p(\mathbf{x}, \mathbf{y}) \, d\mathbf{y}$$

The conditional distribution $p(\mathbf{x} | \mathbf{y})$, which describes the distribution of $\mathbf{x}$ given that we have observed $\mathbf{y}$, is defined through the fundamental relationship:

$$p(\mathbf{x} | \mathbf{y}) = \frac{p(\mathbf{x}, \mathbf{y})}{p(\mathbf{y})}$$

These concepts will recur throughout our exploration of generative models, as we shall see when we examine latent variable models and the applications of Bayes' theorem.

Maximum Likelihood Estimation

Given a parametric family of distributions $p_\theta(\mathbf{x})$, how should we choose the parameters $\theta$ based on our observed data $\mathcal{D}$? The principle of maximum likelihood estimation (MLE) provides an answer with both theoretical justification and practical utility. The intuition is straightforward: we should choose parameters that make the observed data most probable.

Formally, the likelihood of the parameters given the data is the probability (density) of observing our specific dataset under the model:

$$\mathcal{L}(\theta; \mathcal{D}) = \prod_{i=1}^{m} p_\theta(\mathbf{x}_i)$$

This product can become numerically unstable for large datasets, so we typically work with the log-likelihood:

$$\ell(\theta; \mathcal{D}) = \sum_{i=1}^{m} \log p_\theta(\mathbf{x}_i)$$

The maximum likelihood estimate is then:

$$\theta_{\text{MLE}} = \arg\max_\theta \ell(\theta; \mathcal{D}) = \arg\max_\theta \sum_{i=1}^{m} \log p_\theta(\mathbf{x}_i)$$

Maximizing the log-likelihood is equivalent to minimizing the Kullback-Leibler (KL) divergence between the data distribution and the mod