What is Generative AI? Definitions, History, and Scope

Generative artificial intelligence (generative AI) constitutes a paradigm shift in computational creativity, enabling machines to produce novel, coherent content across diverse modalities such as images, text, audio, and video. Unlike discriminative models that map inputs to predefined labels or values, generative models learn the underlying probability distribution of the data, allowing them to sample new instances that resemble the training set. This chapter establishes the foundational context for understanding generative AI by defining its core concepts, tracing its historical evolution from early statistical methods to contemporary deep learning architectures, and outlining the scope of this book. We will examine key milestones—including the advent of generative adversarial networks (GANs), diffusion models, and multimodal transformers—that have propelled the field forward, and set expectations for the reader’s journey through subsequent chapters.

Definition and Core Concepts

At its essence, generative modeling is a framework for learning the joint probability distribution $p(x, y)$ or the marginal distribution $p(x)$ of observed data. The primary objective is to capture the inherent structure, variations, and dependencies within the data so that new samples can be generated by drawing from the learned distribution. Formally, given a dataset $\{x^{(1)}, x^{(2)}, \dots, x^{(N)}\}$ assumed to be drawn from an unknown true distribution $p_{\text{data}}(x)$, a generative model defines a parametric distribution $p_{\text{model}}(x; \theta)$ and adjusts its parameters $\theta$ to minimize the divergence between $p_{\text{model}}$ and $p_{\text{data}}$. Common divergence measures include the Kullback-Leibler (KL) divergence $D_{\text{KL}}(p_{\text{data}} \| p_{\text{model}})$ or the Jensen-Shannon divergence.

This contrasts sharply with discriminative modeling, which directly learns the conditional probability $p(y|x)$ or a decision function $f(x) \to y$, focusing on boundary delineation between classes without concern for how the data is distributed. To illustrate, consider a spam classification task: a discriminative model such as logistic regression learns the probability that an email is spam given its features (e.g., word frequencies), whereas a generative model like naive Bayes would model the joint distribution of features and spam label, enabling it to generate synthetic email texts that mimic real spam. The generative approach thus provides a richer understanding of the data generation process, albeit often at the cost of predictive accuracy on pure classification tasks.

The distinction between generative and discriminative paradigms is not merely academic; it informs model design, evaluation metrics, and application scenarios. Generative models excel at tasks requiring data synthesis, imputation of missing values, and uncertainty quantification through probabilistic sampling. Discriminative models dominate in scenarios where the sole goal is accurate prediction, such as image classification or speech recognition. However, modern deep learning has blurred this dichotomy: many state-of-the-art systems incorporate both generative and discriminative components. For instance, generative adversarial networks (GANs) pit a generator (generative) against a discriminator (discriminative) in a minimax game, while diffusion models employ a denoising network that can be viewed as discriminative (predicting noise) within a generative sampling process.

A central concept in generative modeling is the latent space—a lower-dimensional representation that captures the salient factors of variation in the data. Models such as variational autoencoders (VAEs) and GANs learn to map data points to and from this latent space, enabling controlled generation by interpolating between latent codes. For example, in face generation, moving along a latent dimension might smoothly transition from a smiling face to a neutral one, demonstrating the model’s grasp of semantic attributes. The quality of a generative model is often evaluated by its ability to produce diverse, high-fidelity samples and by the coherence and interpretability of its latent space.

Another foundational idea is density estimation: estimating the probability density function $p(x)$ from data. Explicit density models like autoregressive flows (e.g., PixelCNN) allow exact likelihood computation but may be computationally intensive. Implicit models like GANs generate samples without explicitly defining $p(x)$, trading statistical rigor for sample quality. Understanding these trade-offs is crucial for selecting appropriate models for specific applications.

Historical Evolution: From Statistics to Deep Learning

The quest to model and generate data has deep roots in statistics and probability theory, evolving through several distinct eras marked by theoretical breakthroughs and computational advances.

Early Statistical Methods (1950s–1980s)

The earliest generative approaches were rooted in parametric statistics. Bayesian networks, introduced in the 1980s, provided a graphical framework for modeling joint probability distributions over variables using conditional dependencies. Gaussian mixture models (GMMs) offered a flexible way to model complex data distributions as a weighted sum of Gaussian components, enabling tasks like clustering and density estimation. Hidden Markov models (HMMs) became the backbone of generative sequence modeling in speech recognition, capturing temporal dynamics through latent states. These methods shared limitations: they required strong assumptions about data structure (e.g., linearity, Gaussianity) and struggled with high-dimensional data like images due to the curse of dimensionality.

Neural Networks and the Rise of Representation Learning (1990s–2000s)

The resurgence of neural networks in the late 20th century, particularly with the backpropagation algorithm, opened new avenues for generative modeling. Autoencoders (1980s, popularized in the 2000s) learned efficient data codings by training a neural network to reconstruct its input after passing through a bottleneck layer, implicitly capturing a latent representation. However, early autoencoders produced blurry reconstructions and lacked a principled probabilistic interpretation. Restricted Boltzmann machines (RBMs), a type of energy-based model, introduced the idea of learning stochastic binary features through contrastive divergence, enabling pre-training of deep belief networks (Hinton et al., 2006). This period saw the first glimmers that neural networks could learn hierarchical feature representations, but generating high-quality samples remained elusive.

Deep Generative Models (2013–Present)

The modern era of generative AI began with the fusion of deep learning and probabilistic modeling. In 2013, variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014) provided a scalable, differentiable framework for learning latent variable models by introducing a reparameterization trick that enabled gradient-based optimization. VAEs could generate coherent samples and learn structured latent spaces, but their outputs were often blurry due to a focus on likelihood maximization.

The breakthrough that captured global attention came in 2014 with generative adversarial networks (GANs) (Goodfellow et al., 2014). GANs framed generation as a game between a generator network that synthesizes data and a discriminator network that tries to distinguish real from fake. This adversarial setup yielded remarkably sharp and realistic images but suffered from training instability, mode collapse, and sensitivity to hyperparameters. Over the following years, variants like DCGAN, WGAN, and StyleGAN progressively improved stability, quality, and controllability.

Simultaneously, autoregressive models like PixelCNN (van den Oord et al., 2016) and PixelRNN modeled i