Why Data Is the New Code: The Rise of Data-Centric AI

Why Data Is the New Code: The Rise of Data-Centric AI

Imagine two students preparing for a math competition. The first memorizes ten thousand practice problems, but the answers in the back of the book are sometimes wrong, and a few "problems" are actually jokes from the author's kid, scribbled in the margin. The second student works through one hundred carefully chosen problems, each with clearly written solutions, then redoes the ones she got wrong until she gets them right every time. Both students are equally smart. Which one scores higher on the real test?

If you answered "the second one," you have just predicted the biggest shift in artificial intelligence research of the past five years. Replace "math competition" with "look at a photo and tell me what is in it," and "student" with "neural network," and you have the central claim of this book: in modern machine learning, the quality and structure of the training data matters more than the cleverness of the algorithm.

That is a strange claim. For most of computing history, we believed the opposite. We believed that progress came from smarter algorithms: faster sorts, better compression, more elegant proofs. The data was a fixed input — a benchmark like MNIST or ImageNet that everyone used. The work of the field was to come up with a better model that scored a higher number on that fixed set. If you want to understand why the industry has flipped, and what that flip means for someone starting out today, this chapter is the doorway. The rest of the book is the workshop behind it.

The shift is not subtle. As recently as 2018, talks at the largest machine learning conference, NeurIPS, were overwhelmingly about new architectures: attention mechanisms, residual connections, layer normalization. As recently as 2020, papers about "data" mostly talked about scraping more of it from the web. By 2024, almost every major industry lab had reorganized its teams around data — its quality, its provenance, its labeling, its licenses, its synthetic generation. Andrej Karpathy, who directed AI at Tesla for five years, put it bluntly: "the data, not the model, is the product."

Here is the question this chapter will answer: why has data become the most valuable resource in modern AI, and what does that mean for someone starting out today? By the end of this chapter, you will be able to articulate the data-centric AI thesis, name four companies that won because of data rather than architecture, and run a small experiment showing that one thousand clean examples can outperform one million noisy ones. You will also see the role you can play even before you have any programming skill, which is the second-biggest surprise of this new world.

1.1 The Algorithm Era (1990s–2010): When Mathematics Was King

To see why data-centric AI is a big deal, you have to understand what came before. From roughly 1990 to 2010, the dominant paradigm in machine learning was *algorithm-centric*: the dataset was treated as a given, and the work was to invent a better function for mapping inputs to outputs.

The classic example is the support vector machine, or SVM. In the 1990s, SVMs were the crown jewel of pattern recognition. An SVM tries to draw a clean line (or, in higher dimensions, a clean hyperplane) that separates two classes of data — say, "spam" and "not spam." The line is chosen to maximize the margin, the empty space between the two groups. The math behind this is elegant: it uses a trick called the kernel method to handle non-linear boundaries without ever having to compute the transformed coordinates explicitly. Researchers loved SVMs because the math was beautiful, the training was efficient, and the resulting model had strong theoretical guarantees. Vladimir Vapnik and Corinna Cortes's 1995 paper introducing the soft-margin SVM is still studied in graduate courses today.

Around the same time, ensemble methods were rising. Random forests, popularized by Leo Breiman in 2001, combine hundreds of decision trees, each trained on a random subset of the data, and average their votes. Gradient

boosting, invented by Yoav Freund and Robert Schapire, took a different tack: it built trees one at a time, each new tree correcting the mistakes of all the previous ones combined. Both methods won competitions, both methods produced papers, and almost none of the breakthroughs came from "more data." They came from mathematical cleverness.

Even the era's benchmarks reflected this mindset. MNIST, the handwritten digit dataset, has only seventy thousand tiny grayscale images. CIFAR-10 has sixty thousand small color photos across ten classes. ImageNet, released in 2009, was the first dataset large enough to make the field uncomfortable, but even it was treated as a fixed yardstick: the goal was to climb the leaderboard, not to ask whether the yardstick itself was wrong. The data was the weather; the algorithm was the meteorologist.

If you are a teenager reading this, you have probably never experienced that world. The phone in your pocket runs a voice assistant trained on tens of millions of hours of audio. The recommended video you just watched was selected from among hundreds of millions by a model that was retrained overnight. The algorithms powering those systems — mostly variants of the transformer, which we will meet in Chapter 2 — are not fundamentally new inventions of the 2020s. The transformer was published in 2017. What changed was the *scale* and *quality* of the data fed into them.

1.2 The Data Wall

By the early 2020s, the biggest labs in the world had hit a problem they did not expect. They had thrown more data at bigger models, and the gains had started to flatten.

The story begins with a simple question: how many parameters does a language model need to be good at conversation? For years, the answer seemed to be "as many as you can afford." OpenAI's GPT-2, released in 2019, had 1.5 billion parameters. GPT-3, in 2020, had 175 billion. That is a hundredfold jump, and the qualitative leap was enormous — suddenly the model could write passable essays, summarize articles, and even produce working code. The temptation, naturally, was to do it again. Train a model with ten trillion parameters. Surely it would be ten times better.

It was not. The DeepMind team behind Chinchilla, a 70-billion-parameter language model released in 2022, ran a careful study. They trained models of different sizes on different amounts of data and plotted the result. The plot looked like a set of curves bending downward: once a model had "seen" enough tokens — the rough equivalent of words and word-pieces — additional data stopped helping. Worse, models that were smaller but trained on *more* data consistently outperformed larger models trained on *less* data. The lesson was uncomfortable: the bottleneck was no longer the size of the model. It was the size, and quality, of the dataset.

This is the data wall, and it has three faces.

The first face is scarcity. The web has, optimistically, a few hundred billion words of useful English text. That sounds like a lot, but most modern language models are trained on tokens that include code, multilingual text, scientific papers, and books. After deduplication and filtering, the genuinely *informative* tokens run out. Common Crawl, the giant scraped archive that powers much of this work, is heavily polluted with spam, boilerplate, and machine-generated text of dubious quality. If you train on the polluted version, you get a model that hallucinates more, follows instructions less reliably, and refuses less often when it should refuse.

The second face is cost. Training GPT-3 reportedly cost on the order of ten million dollars in compute. Training a next-generation model at the same ratio of data-to-parameters would cost a hundred million or more. But the marginal returns shrink: doubling the compute might improve benchmark scores by a single percentage point. At some point, th