Foundations of Generative Modeling: Probability, Information, and Learning Paradigms

At the heart of generative artificial intelligence lies a deceptively simple idea: to learn the underlying probability distribution of data so that we can create new, plausible samples from it. This chapter establishes the essential mathematical and conceptual scaffolding required to understand how generative models work. We will journey from the fundamental language of probability, through the quantitative tools of information theory, to the critical distinction between *generative* and *discriminative* modeling paradigms. By the end, you will not only grasp the "what" and "why" of generative modeling but also have a practical, code-aided intuition for its core mechanics, setting a firm foundation for the deep dive into specific architectures like GANs and diffusion models in subsequent chapters.

1.1 Probability Theory: The Language of Uncertainty

Generative modeling is, at its core, the task of probability estimation. Before we can model complex data like images or text, we must be fluent in the basic grammar of probability theory.

1.1.1 Random Variables and Probability Distributions

A random variable $X$ is a variable whose possible values are outcomes of a random phenomenon. We describe the behavior of $X$ through its probability distribution. Distributions fall into two primary categories:

• **Discrete Distributions:** For variables with a countable set of outcomes (e.g., the result of a coin flip). The **Probability Mass Function (PMF)** $P(X = x)$ gives the probability of each exact value $x$.
• **Continuous Distributions:** For variables over a continuous range (e.g., height, pixel intensity). We use a **Probability Density Function (PDF)** $p(x)$. The probability that $X$ falls within an interval $[a, b]$ is the area under the PDF curve: $P(a \leq X \leq b) = \int_a^b p(x) dx$. Crucially, $p(x)$ itself is not a probability; it is a *density*.

Example 1.1: The Bernoulli Distribution The simplest non-trivial distribution models a single binary outcome (e.g., heads/tails, success/failure). If $X \sim \text{Bernoulli}(p)$, then: $$ P(X = 1) = p, \quad P(X = 0) = 1-p $$ Here, $p$ is the probability of "success" (e.g., heads). The PMF is discrete.

Example 1.2: The Univariate Gaussian (Normal) Distribution This is the workhorse of continuous probability. If $X \sim \mathcal{N}(\mu, \sigma^2)$, its PDF is: $$ p(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right) $$ The parameters $\mu$ (mean) and $\sigma^2$ (variance) completely define the distribution's center and spread. Its symmetric, bell-shaped curve is ubiquitous because of the Central Limit Theorem, which states that the sum of many independent random variables tends toward a Gaussian distribution regardless of their original distributions.

```python import numpy as np import matplotlib.pyplot as plt from scipy.stats import bernoulli, norm

--- Bernoulli Example ---

p = 0.6 x_bernoulli = [0, 1] pmf = [1-p, p] plt.figure(figsize=(10, 4)) plt.subplot(1, 2, 1) plt.bar(x_bernoulli, pmf, tick_label=['0 (Tails)', '1 (Heads)'], color='skyblue') plt.title(f'Bernoulli PMF (p={p})') plt.ylabel('Probability')

--- Gaussian Example ---

mu, sigma = 0, 1 x_gauss = np.linspace(mu - 4*sigma, mu + 4*sigma, 1000) pdf = norm.pdf(x_gauss, mu, sigma) plt.subplot(1, 2, 2) plt.plot(x_gauss, pdf, 'b-', lw=2) plt.fill_between(x_gauss, pdf, alpha=0.3) plt.title(f'Gaussian PDF (μ={mu}, σ={sigma})') plt.xlabel('x') plt.ylabel('Density') plt.tight_layout() plt.show() ```

1.1.2 Joint, Marginal, and Conditional Distributions

Real-world data is often multi-dimensional. If we have two random variables $X$ and $Y$ (e.g., height and weight), their joint distribution $P(X=x, Y=y)$ or $p(x,y)$ describes the probability of them taking specific values simultaneously.

From the joint distribution, we derive:

• **Marginal Distribution:** The distribution of one variable ignoring the other. For discrete variables: $P(X=x) = \sum_y P(X=x, Y=y)$. For continuous: $p(x) = \int p(x,y) dy$.
• **Conditional Distribution:** The distribution of one variable given the value of another. **Bayes' Theorem** is the fundamental rule for reversing conditional probabilities:

$$ P(Y|X) = \frac{P(X|Y) P(Y)}{P(X)} $$ For densities: $p(y|x) = \frac{p(x|y) p(y)}{p(x)}$. Here, $P(Y)$ or $p(y)$ is the prior (our belief before seeing $X$), $P(Y|X)$ or $p(y|x)$ is the posterior (updated belief after seeing $X$), and $P(X|Y)$ or $p(x|y)$ is the likelihood (probability of the evidence given the hypothesis).

Example 1.3: A Simple Medical Test Let $D$ be the event "has disease" ($P(D)=0.01$). Let $T$ be the event "tests positive". Suppose the test is 99% sensitive ($P(T|D)=0.99$) and 95% specific ($P(\neg T|\neg D)=0.95$, so $P(T|\neg D)=0.05$). What is the probability a person has the disease given a positive test? Bayes' Theorem: $$ P(D|T) = \frac{P(T|D) P(D)}{P(T)} = \frac{0.99 \times 0.01}{P(T)} $$ We need $P(T)$, the total probability of a positive test: $$ P(T) = P(T|D)P(D) + P(T|\neg D)P(\neg D) = (0.99 \times 0.01) + (0.05 \times 0.99) \approx 0.0594 $$ Thus, $$ P(D|T) \approx \frac{0.0099}{0.0594} \approx 0.1667 $$ Despite the test's high accuracy, because the disease is rare (low prior), a positive result still means only a ~16.7% chance of having the disease. This counterintuitive result highlights the power—and necessity—of Bayesian reasoning.

1.1.3 Independence and Conditional Independence

Two variables $X$ and $Y$ are independent if knowing one gives no information about the other: $P(X,Y) = P(X)P(Y)$ or $p(x,y) = p(x)p(y)$. This implies $P(Y|X) = P(Y)$.

They are conditionally independent given a third variable $Z$ if: $$ P(X,Y|Z) = P(X|Z) P(Y|Z) $$ This is a crucial concept in graphical models and many generative architectures, where complex dependencies can be factored via latent variables $Z$.

---

1.2 Information Theory: Quantifying Information and Distinction

While probability provides the framework, information theory gives us the tools to *measure* and *compare* probability distributions—the core task of generative modeling.

1.2.1 Shannon Entropy

Entropy $H(X)$ is the expected amount of "surprise" or information contained in the outcomes of a random variable $X$. For a discrete distribution: $$ H(X) = -\sum_{x} P(x) \log P(x) $$ For a continuous distribution, we use differential entropy: $$ h(X) = -\int p(x) \log p(x) dx $$ Key properties:

• Entropy is always non-negative for discrete variables (with log base 2, units are **bits**).
• It is maximized for a uniform distribution (maximum uncertainty).
• For a Gaussian $\mathcal{N}(\mu, \sigma^2)$, the differential entropy is $h(X) = \frac{1}{2} \log(2\pi e \sigma^2)$, which increases with variance $\sigma^2$ (more spread = more uncertainty).

Entropy quantifies the inherent randomness in a single distribution.

1.2.2 Kullback–Leibler (KL) Divergence

The KL divergence $D_{\text{KL}}(P \| Q)$ measures how one probability distribution $P$ diverges from a second, reference distribution $Q$. It is not a true distance (it's asymmetric), but it is fundamental: $$ D_{\text{KL}}(P \| Q) = \sum_{x} P(x) \log \frac{P(x)}{Q(x)} \quad \text{(discrete)} $$ $$ D_{\text{KL}}(P \| Q) = \int p(x) \log \frac{p(x)}{q(x)} dx \quad \text{(continuous)} $$ Interpretation: It is the expectation (under $P$) of the difference in log-probabilities between $P$ and $Q$. It answers: "If I design a code assuming data comes from $Q$, but it actually comes from $P$, how many extra bits per data point will I need on average?"

Critical Properties: 1. Asymmetry: $D_{\text{KL}}(P \| Q) \neq D_{\text{KL}}(Q \| P)$. $D_{\text{KL}}(P \| Q)$ is the "information loss" when using $Q$ to approximate $P$. Minimizing $D_{\text{KL}}(P_{\text{data}} \| P_{\text{model}})$ is a common generative modeling objective (e.g., in Variational Autoencoders). 2. **Non