The Day the Data Ran Out

Chapter 1: The Day the Data Ran Out

The conference room in San Francisco was the kind of place that had been designed to make people feel like the future had already arrived. Floor-to-ceiling glass. A long oak table polished to a mirror. Outside, the October fog was doing its slow theatrical crawl up the Salesforce Tower, but inside, the air was warm and the espresso was strong and forty of the most important people in artificial intelligence were leaning forward in their chairs, listening to a young engineer with a faded band T-shirt explain, in a careful, almost apologetic voice, that she had just discovered something that frightened her.

Her name was Mira Okafor. She was a data engineer, not a researcher, not a founder, not a person whose name appeared in the splashy magazine profiles that the industry loved to commission about itself. She was the kind of person who actually built the pipelines that fed the great machines, the person who sat in basements and argued with cloud-billing departments and stayed late at night sorting out the unglamorous plumbing of intelligence. She had been asked, on very short notice, to present at an off-the-record summit that had been convened by a coalition of frontier labs because they were running out of something, and they were not sure what to do about it.

What they were running out of was data.

"I want to show you a number," Mira said, clicking to her next slide. The number was 380. "That's the number of petabytes of clean, deduplicated, high-quality English text that exists on the open web. Not the surface web, not the dark web, but the part of the internet that is publicly crawlable, that is not behind a login wall, that is not generated by machines, that is not a copy of a copy of a copy. Three hundred and eighty petabytes. That's the upper bound. If we threw out everything else, if we stripped out the spam and the duplicates and the SEO sludge and the product descriptions, three hundred and eighty petabytes is roughly the total stock of human-written English that we have ever been able to find."

The room was quiet. Mira clicked again. The next number was 5,600. "That's the number of petabytes of human-written English text that the largest training runs in 2024 had effectively consumed, including duplicates and lower-quality material. We have already trained on more text than exists. Our models have, in many cases, seen the same web page hundreds of times. We are not running out of data. We ran out of data a long time ago. The question is what we do next."

This is the moment that this book is built around. Not the moment Mira walked off the stage. Not the moment the executives in the room stopped taking notes and started looking at each other. The moment is the one we are living in now, the slow, strange, slightly disorienting moment when the technology that had felt inexhaustible, that had for ten years behaved as if its only limits were our ambition and our electricity bills, ran up against a much more ordinary limit. A wall that was not made of silicon or regulation or even money, but of words, and images, and audio, and the basic fact that human beings had only ever made so much of them.

The argument of this book is simple, and it is uncomfortable. Artificial intelligence, the engine of pattern recognition that has begun to feel like something close to a partner or a rival or a god, has an engine of its own, and that engine is running low on fuel. The fuel is data. The pipeline is broken. And the people who built it are now, with various degrees of panic and various degrees of honesty, scrambling to figure out how to keep the lights on.

To understand how we got here, you have to understand what AI actually is, not the marketing version, not the version that lights up your phone or fills your screen with plausible text, but the version that lives in the data centers, that lives in the spreadsheets of the people who build it, that lives in the unglamorous plumbing that Mira and her colleagues spend their days working on. At its core, an AI model is a compression algorithm trained on examples. You show it a billion pictures of cats, and it learns the statistical shape of "cat." You show it a trillion tokens of text, and it learns the statistical shape of language. The intelligence, such as it is, is downstream of the examples. Take away the examples and you have a very expensive random number generator.

For most of the last decade, the examples felt infinite. The web was vast, and it was growing, and the people scraping it felt like they were taking water from an ocean. They could afford to be wasteful. They could afford to download the same page a hundred times, to keep the spam and the duplicates, to throw petabytes at a problem that might have been solvable with gigabytes, because nobody had a reason to be careful. The whole enterprise felt like a gold rush, and gold rushes are not known for their attention to long-term sustainability.

But the web is not an ocean. The web is a finite reservoir. Human beings, it turns out, can only write so much. We can only take so many photographs. We can only record so many hours of speech. And we can only publish so much of it in places where scrapers can reach. The total stock of human-written text on the open internet is, by the most generous estimates, somewhere between 300 and 500 petabytes of unique, deduplicated, machine-readable content. That is a lot of text. It is also, when you do the math on what the next generation of models will need, very little.

This chapter is the story of how the people who run the AI industry came to understand that, and what they are doing about it. We are going to meet a researcher at Common Crawl, the nonprofit that has quietly become one of the most important data sources in the history of computing, who realized that nearly every public webpage has now been harvested several times. We are going to watch a copyright lawyer explain why the landmark lawsuits now grinding through the courts are not just a legal problem but a logistical one. We are going to sit in a research lab in Zürich with a climate scientist who has done the math on what it would cost, in electricity and in water, to train the next generation of frontier models. And we are going to follow Mira, our guide for this whole book, as she tries to answer the unglamorous question that the industry is now obsessing over: if we cannot get more data, what do we do with the data we already have?

But before any of that, we need to understand the shape of the problem, because the shape of the problem is the argument of the book. Let me draw it for you.

```mermaid flowchart TD subgraph SUPPLY["Data Supply: Finite and Depleting"] S1[Public Web Text<br/>~380 PB] S2[Books and Journals<br/>~20 PB] S3[Image and Video Archives<br/>~5,000 PB but heavily duplicated] S4[Speech and Audio<br/>~50 PB] S1 --> S5[First Pass: Harvested] S2 --> S5 S3 --> S5 S4 --> S5 S5 --> S6[Second Pass: Marginal Returns] S6 --> S7[Third Pass: Saturation] S7 --> S8[Bottleneck: No New Human Data] end

subgraph DEMAND["Compute Demand: Exponential"] D1[2018: GPT-1<br/>0.1 PF-days] D2[2020: GPT-3<br/>3,600 PF-days] D3[2023: GPT-4<br/>~50,000 PF-days] D4[2025: Frontier Models<br/>~500,000 PF-days] D5[2027?: Next Generation<br/>~5,000,000 PF-days] D1 --> D2 D2 --> D3 D3 --> D4 D4 --> D5 end

S8 --> BOT[THE BOTTLENECK] D5 --> BOT BOT --> P1[Path A: Clean and Reuse] BOT --> P2[Path B: Generate Synthetic Data] BOT --> P3[Path C: New Sensors and Modalities] ```

The diagram is doing a lot of work, and it is worth slowing down on it. On the left, you have the supply side. It is a stack, and the stack is finite. Even if we add every public webpage, every library, every radio archive, every surveillance camera