What Is NLP and Why Should You Care?

What Is NLP and Why Should You Care?

A machine that can read every text message ever sent. A system that translates a hundred languages in real time. A chatbot that does not sound like a robot. None of these existed for most of human history, and all of them are powered by the same field: natural language processing. This chapter answers the question at the heart of this book: how can a machine understand language, where did that capability come from, and why does it matter to you right now, today, as a teenager living through the most linguistically saturated era in human history.

The technology you are about to study is not a single tool. It is a stack of ideas layered on top of each other for more than seventy years, with each layer fixing the failure modes of the one below. By the time you finish this chapter, you will be able to read almost any modern NLP article, identify which era its techniques come from, and explain the field to a friend who has never heard of it. The rest of the book is the hands-on part. This chapter is the map.

The Central Question: How Can a Machine Understand Language?

Consider what happens in the two seconds after you type "best pizza near me" into Google. The string you typed is twelve characters long, including a space. A silicon chip in a data center somewhere on the planet receives those twelve characters as a sequence of bytes. From that sequence, the system produces a ranked list of pizzerias in your city, often within a fraction of a second. Somehow, abstract meaning traveled from your brain, through your fingers, into a network of wires and transistors, and back out as a useful answer. The bridge that makes that trip possible is what this book is about.

The phrase "understand language" hides more than it reveals. When you read the sentence "Time flies like an arrow," your brain instantly performs a small miracle. You parse the syntax, retrieve the idiomatic meaning, visualize an arrow in flight, and probably do not even notice the alternative reading: "time flies [the insects] in the way that an arrow flies." That second reading is grammatically valid and semantically possible, but you suppress it automatically because the idiomatic reading is ten thousand times more likely given the rest of the world you have experienced. A computer has none of that. It has numbers. Every achievement in natural language processing is, at some level, a story about converting the messy, ambiguous, context-rich structure of human language into numbers a machine can manipulate, and then converting the results back into something a human finds useful.

The central technical problem can be stated simply: language is symbolic, computers are numeric. Words are discrete symbols with no inherent mathematical relationship. "Cat" is not closer to "dog" than to "banana" in any obvious numerical sense, yet every English speaker knows that "cat" and "dog" are more alike than "cat" and "banana." The job of NLP is to invent the math that makes the obvious true for a machine.

A useful first experiment, which you can do right now without any code, is to list all the meanings of the word "set" as you can. Native English speakers usually produce 30 to 60 distinct meanings in a few minutes. They include: a set of objects (collection), to set a clock (configure), to set something down (place), a tennis set (round), set as in "set in stone" (fixed), a set expression (idiom), to set a price (assign), to set a bone (rearrange), and many more. The single word "set" is doing the work of dozens of different concepts, and a human reader picks the right one based on context. A machine has to do the same disambiguation, and it is far harder for the machine than for you, because the machine has not lived a lifetime of setting bones and tennis sets and clocks.

A second experiment: take the sentence "I saw the man with the telescope." In English, "with the telescope" can attach to "saw" (I used the telescope to see the man) or to "man" (I saw the man who had a telescope). This is the famous "low attachment" vs. "high attachment" ambiguity. Linguists have known about it for decades. NLP systems have to resolve it, often by using statistical preferences: in a large corpus, low attachment is more common, so a good NLP system defaults to "I used the telescope." But in a story about astronomers, high attachment is more likely, and a contextual system should pick that instead. The system has to know that astronomers carry telescopes, even though the sentence does not say so explicitly.

A third experiment, even more striking: the sentence "The trophy did not fit in the suitcase because it was too small." Most readers, on first reading, assume "it" refers to the suitcase. The trophy did not fit because the suitcase was too small. But on reflection, "it" could refer to the trophy, and the sentence is still grammatical. The same is true of "The trophy did not fit in the suitcase because it was too big." Now "it" more naturally refers to the trophy. These are examples of Winograd schemas, named after the linguist Terry Winograd, and they are considered a hard test of language understanding. Until 2018 or so, no NLP system could reliably solve them. Modern transformer-based systems can solve many of them, but not all. The fact that this is still an open problem should give you a sense of how difficult language is for a machine, and how impressive the recent progress has been.

This problem is not new. The phrase "artificial intelligence" was coined in 1956, and the idea of a machine that could read and write has been a touchstone of the field ever since. What has changed is the answer. For most of the field's history, the answer was: hand-coded rules, then statistical patterns, then neural networks, then transformers. We will trace all of those answers in this chapter. By the end, you will have a working mental map of where NLP came from, where it is right now, and what tools you will be using to build with it for the rest of this book.

Defining NLP: The Meeting Point of Three Disciplines

Natural language processing is a subfield of artificial intelligence, but unlike most AI subfields, it is the meeting point of three distinct academic traditions, each of which is a century or more old.

Linguistics supplies the structural theory. Linguists have spent more than a century cataloging how language works. Phonology is the study of the sounds that make up words, and the rules by which they combine. Morphology is the study of how words are built from smaller pieces (the word "unbreakable" is morphologically a negation, a root, and a suffix, in that order). Syntax is the study of how words combine into phrases and sentences, governed by grammar rules that are largely unconscious. Semantics is the study of what sentences mean, independent of context. Pragmatics is the study of what speakers mean in context: sarcasm, implication, indirect speech acts. Without linguistics, NLP would be guesswork. With it, you can decompose the problem: an NLP system that understands syntax can be designed differently from one that only needs to detect sentiment, and the difference is the difference between a fragile system and a robust one.

Computer science supplies the algorithms and the hardware. Search algorithms, data structures, complexity analysis, distributed systems: these are the tools that turn linguistic ideas into running code. Modern NLP runs on GPUs and TPUs originally designed for video games and graphics rendering, and the algorithms that dominate the field, transformers in particular, are deeply tied to the realities of parallel hardware. A transformer runs efficiently on a GPU because it does the same matrix multiplication thousands of times in parallel. The same algorithm on a CPU, with its serial architecture, would be far slower. The hardware shapes the algorithms, and the algorithms shape what is possible.

Machine learning supplies the data-dr