Chapter 1: What is Generative AI? History and Core Concepts

Generative artificial intelligence (AI) represents a fundamental shift in the objectives of machine learning. While traditional discriminative models learn to map inputs to outputs—such as classifying an image as a cat or dog—generative models strive to understand and replicate the very process that produced the data. At its core, generative AI learns the probability distribution $p_{data}(x)$ over a dataset of samples $x$. Once this distribution is approximated, new samples can be drawn from it, creating data that is statistically similar to the training set yet entirely novel. This capability to "create" rather than just "recognize" opens up a universe of applications, from designing virtual worlds to discovering new drugs. The journey from early statistical methods to modern deep generative models has been marked by revolutionary ideas, each building upon the last to achieve increasingly realistic and diverse generations. This chapter lays the groundwork by defining key concepts, tracing historical milestones, surveying major applications, and explaining why generative AI is a transformative technology. We conclude with a motivating example—generating a photorealistic human face—to illustrate the process in action and set the stage for the technical deep dive that follows.

Generative vs. Discriminative: A Fundamental Dichotomy

To grasp generative AI, one must first distinguish it from its more common counterpart: discriminative modeling. Discriminative models, such as support vector machines, logistic regression, and most convolutional neural networks used for classification, learn the conditional probability distribution $p(y|x)$. Given an input $x$ (e.g., an image), they predict the label $y$ (e.g., "cat" or "dog"). Their objective is to find a decision boundary that separates classes as accurately as possible. They do not concern themselves with how the data itself was generated; they only care about the mapping from inputs to outputs.

Generative models, in contrast, aim to learn the full joint distribution $p(x, y)$ or, in unsupervised scenarios, the marginal distribution $p(x)$. By modeling $p(x)$, a generative model can produce new $x$ values that resemble the training data. This distinction is not merely academic—it leads to different capabilities and limitations. Discriminative models excel at tasks where the goal is prediction or classification, often achieving higher accuracy on these specific tasks because they focus all their capacity on the decision boundary. Generative models, however, offer a deeper understanding of the data structure and enable synthesis, which is essential for creative applications, data augmentation, and simulation.

Mathematically, for a dataset $\{x^{(i)}\}_{i=1}^N$ drawn from an unknown true distribution $p_{data}(x)$, a generative model defines a parametric family of distributions $p_\theta(x)$ with parameters $\theta$. The training objective is typically to minimize the divergence between $p_\theta$ and $p_{data}$. The most common objective is maximum likelihood estimation (MLE):

$$ \theta^* = \arg\max_\theta \mathbb{E}_{x \sim p_{data}} [\log p_\theta(x)]. $$

In practice, we approximate the expectation with the empirical average over the training set:

$$ \theta^* \approx \arg\max_\theta \frac{1}{N} \sum_{i=1}^N \log p_\theta(x^{(i)}). $$

However, directly computing $\log p_\theta(x)$ is intractable for most complex models, leading to various approximations and alternative objectives, as we will see in later chapters.

Consider a concrete example: handwritten digit recognition. A discriminative model like a softmax classifier takes an image $x$ and outputs a probability distribution over the ten digit classes $y \in \{0,1,...,9\}$. It learns $p(y|x)$. A generative model, such as a variational autoencoder (VAE), instead learns to generate new images of digits. It models $p(x)$ by introducing a latent variable $z$ (e.g., a low-dimensional representation of digit style) and learns $p(x|z)$ and $p(z)$. After training, we can sample $z \sim p(z)$ and then $x \sim p_\theta(x|z)$ to produce a new digit image. The generative model captures not only which digits exist but also their variations in stroke thickness, slant, and other attributes encoded in $z$.

This dichotomy extends to many domains. In natural language processing, a discriminative model might classify the sentiment of a sentence (positive/negative), while a generative model like GPT generates new sentences. In audio, a discriminative model might detect whether a clip contains speech, while a generative model like WaveNet synthesizes speech waveforms. The generative approach is inherently more ambitious—it seeks to model the entire data manifold—but it is also more challenging and computationally intensive.

Core Concepts of Generative Modeling

Data Distribution and Probability Density Estimation

The central task of generative AI is probability density estimation: given samples from an unknown distribution $p_{data}(x)$, estimate a model distribution $p_\theta(x)$ that closely matches it. The data $x$ can be discrete (e.g., text tokens) or continuous (e.g., pixel values, audio waveforms). For continuous data, $p_\theta(x)$ is a probability density function, and we often work with log-likelihood $\log p_\theta(x)$. For discrete data, it is a probability mass function.

In high-dimensional spaces (e.g., a $256 \times 256$ RGB image has $256 \times 256 \times 3 = 196,608$ dimensions), modeling $p(x)$ directly is extremely difficult because the distribution is highly complex and the data occupies a tiny, low-dimensional manifold within the ambient space. Generative models address this by introducing structure: either through explicit density models that allow tractable computation of $p_\theta(x)$ (e.g., autoregressive models, normalizing flows) or implicit models that define a procedure to generate samples without explicitly computing the density (e.g., GANs, diffusion models).

The quality of a generative model is often measured by how well the generated samples match the real data in terms of diversity and fidelity. However, evaluating density estimation on high-dimensional data is tricky; log-likelihood can be misleading if the model assigns high probability to unrealistic samples. Hence, a suite of metrics has been developed, including Inception Score (IS), Fréchet Inception Distance (FID), and precision-recall curves for generative models, which we will discuss in Chapter 14.

Latent Space and Representation Learning

Most modern generative models rely on a latent variable $z$ that captures the underlying factors of variation in the data. The generative process is typically defined as:

$$ p_\theta(x) = \int p_\theta(x|z) p(z) \, dz, $$

where $p(z)$ is a simple prior distribution (e.g., standard Gaussian $\mathcal{N}(0, I)$), and $p_\theta(x|z)$ is a parameterized function (usually a neural network) that maps latent codes to data space. This formulation decomposes the difficult task of modeling $p(x)$ into two parts: learning a meaningful latent space $z$ and learning the decoder $p_\theta(x|z)$.

The latent space is expected to be semantically meaningful and structured. For example, in a face generator, interpolating between two latent codes $z_1$ and $z_2$ should yield a smooth transition between the corresponding faces, with intermediate codes representing plausible faces. This property is crucial for applications like attribute editing (e.g., changing age or expression) and creative exploration.

Models differ in how they learn the latent representation. Variational autoencoders (VAEs) explicitly enforce a structured latent space by approximating the posterior $p(z|x)$ with a variational distribution $q_\phi(z|x)$ and optimizing the evidence lower bound (ELBO). Generative adversarial networks (GANs) do not explicitly model $p(z)$; instead, the latent space is induced by the mapping from a random noise vector $z \sim \math