The Dream of Synthesis: Pre-GAN Foundations and the Quest for Likelihood

Our odyssey begins not with a bang, but with a careful, probabilistic whisper. In the early 2010s, the field of generative modeling was a quiet garden of probabilistic reasoning, where the dominant paradigm was the gentle art of likelihood maximization. Researchers sought to build models that could truly understand the data distribution, not just mimic it. This pursuit was guided by a single, compelling principle: if a model can assign high probability to the real data, it must have captured the essence of that data. But as we shall see, this noble quest led to a profound dilemma—a trade-off between fidelity and efficiency that would haunt the field until the advent of Generative Adversarial Networks. The landscape was defined by two main families of models: the meticulous, pixel-by-pixel artisans of autoregressive networks, and the compact, latent-space dreamers of variational autoencoders. Both pledged allegiance to the same deity—the likelihood—but each paid a different, heavy tax.

Maximum Likelihood: The Guiding Principle

At the heart of pre-GAN generative modeling lies maximum likelihood estimation (MLE). The goal is simple: given a dataset $\{x^{(i)}\}_{i=1}^N$ assumed to be drawn from an unknown distribution $p_{\text{data}}(x)$, we wish to find a model $p_\theta(x)$ with parameters $\theta$ that makes the observed data most probable. Formally, we maximize the log-likelihood: $$ \log p_\theta(\mathcal{D}) = \sum_{i=1}^N \log p_\theta(x^{(i)}). $$ This objective is intuitively satisfying—it directly measures how well the model fits the data. Moreover, under i.i.d. assumptions, maximizing log-likelihood is equivalent to minimizing the Kullback-Leibler divergence $D_{\text{KL}}(p_{\text{data}} \| p_\theta)$ up to a constant, meaning we are seeking the model distribution closest to the true data distribution in the KL sense. This provided a rigorous, mathematically sound foundation for generative modeling.

However, the elegance of MLE belies a computational chasm. To compute $\log p_\theta(x)$ for a complex, high-dimensional datum like an image, we must integrate over all possible latent variables or sum over all possible configurations—an intractable task for all but the simplest models. The central challenge of the early 2010s was therefore to design models where the likelihood $p_\theta(x)$ could be computed efficiently and exactly, or at least approximated with a tight, tractable lower bound. Two architectural philosophies emerged to meet this challenge, each with its own strengths and inherent limitations.

Autoregressive Models: The Pixel-by-Pixel Symphony

Autoregressive models tackle the intractability problem by factorizing the joint distribution over observed variables (e.g., image pixels) into a product of conditional distributions along a fixed ordering. For an image $x$ with $n$ pixels, we write: $$ p_\theta(x) = \prod_{i=1}^n p_\theta(x_i \mid x_{<i}), $$ where $x_{<i}$ denotes all pixels that precede $x_i$ in the ordering (typically raster scan: left-to-right, top-to-bottom). This decomposition turns the formidable problem of modeling a $n$-dimensional distribution into a sequence of $n$ one-dimensional conditional distributions, each of which can be modeled by a neural network.

PixelRNN: The Recurrent Pioneers

The first major success in this direction was PixelRNN, introduced by van den Oord et al. (2016). PixelRNN uses recurrent neural networks (RNNs) to capture the dependencies between pixels. The key idea is to process the image in a row-by-row manner, using a long short-term memory (LSTM) network that maintains a hidden state summarizing all previously seen pixels. Specifically, for each row, a *Row LSTM* scans the pixels from left to right, with its hidden state initialized from the final state of the previous row. To further capture two-dimensional structure, a *Diagonal BiLSTM* is employed, which processes the image in a diagonal fashion to allow information to flow from both the left and above.

PixelRNN was groundbreaking: it was the first model to achieve state-of-the-art results on image generation benchmarks while providing exact likelihood computation. On the CIFAR-10 dataset, it achieved a negative log-likelihood (NLL) of 3.86 bits per dimension (bits/dim), a significant improvement over previous approaches. The model could be trained by maximizing the exact log-likelihood via stochastic gradient descent, and its performance could be directly monitored using this metric.

But the very factorization that made likelihood tractable imposed a crippling burden on sampling. To generate a new image, one must sample pixels sequentially: start with the top-left pixel, then the next, and so on, each time feeding the newly generated pixel into the RNN to update its state before predicting the next. For a modest $32 \times 32$ image, this requires 1,024 sequential steps; for a $64 \times 64$ image, 4,096 steps. Each step involves a forward pass through the LSTM, which is computationally expensive due to its recurrent nature. The sampling process is inherently sequential and cannot be parallelized across pixels. As a result, generating a single high-resolution image could take several seconds or even minutes on a GPU—a far cry from the instantaneous generation we might desire.

PixelCNN: The Convolutional Revolution

Shortly after PixelRNN, van den Oord et al. introduced PixelCNN, which replaced the recurrent connections with masked convolutional layers. The core innovation is the *masked convolution*: a standard 2D convolution where the filter weights are zeroed out for all input pixels that come after the current pixel in the raster order. This ensures that the prediction for pixel $x_i$ depends only on $x_{<i}$, preserving the autoregressive property while allowing the use of fast, parallelizable convolutional networks.

PixelCNN retains the exact likelihood computation of PixelRNN because the factorization remains identical. However, its convolutional architecture is more efficient during training: since all pixels in an image are processed simultaneously (thanks to the masks, each pixel’s receptive field is restricted to preceding pixels), the training can be fully parallelized across pixels within an image, leading to much faster training times compared to PixelRNN.

Sampling, however, remains stubbornly sequential. Even though the network is convolutional, the generative process still requires generating one pixel at a time and feeding it back into the network to predict the next. The convolutional structure does not change the fundamental sequential dependency; it only changes the computational graph. Thus, the sampling bottleneck persists. PixelCNN achieved similar likelihoods to PixelRNN on CIFAR-10 (around 3.0 bits/dim) but with faster training.

Subsequent variants like PixelCNN++ (Salimans et al., 2017) introduced improvements such as discretized logistic mixture likelihoods for continuous-valued images, residual connections, and dropout, pushing the NLL on CIFAR-10 down to an impressive 2.92 bits/dim. Yet, the sampling speed remained orders of magnitude slower than feedforward models. The message was clear: to get sharp, high-fidelity samples, one had to pay the price of slow, sequential generation.

Variational Autoencoders: The Latent Compromise

While autoregressive models embraced the full complexity of pixel-level dependencies, variational autoencoders (VAEs) took a radically different approach: introduce a low-dimensional latent variable $z$ and model the data distribution as a mixture: $$ p_\theta(x) = \int p_\theta(x \mid z) p(z) \, dz. $$ Here, $p(z)$ is a simple prior (usually a standard normal $\mathcal{N}(0, I)$), and $p_\theta(x \mid z)$ is a decoder network that maps a latent code to a data point. The intuition is appealing: if we can learn a mapping from a structured latent space to data space, we can generate new samples by simply sampling $z \sim p(z)$ and passing it through