The Geometry of Thought: How Machines Learned to Mean

The Geometry of Thought: How Machines Learned to Mean

The Riddle at the Heart of Understanding

Consider a simple sentence: "The cat sat on the mat." Before you even finish reading these words, something remarkable occurs in your mind. You form a mental image—or perhaps several competing images—of a cat, you understand spatial relationships, you grasp causality (the cat's sitting preceded or occurred on the mat), you recognize the definite article "the" as referencing a specific rather than generic cat, and you access a thousand subtle associations: the texture of fur, the sound of purring, the particular way a cat settles its weight when it sits, the cultural connotations of cats in various contexts. Your brain performs this feat instantaneously, effortlessly, and—most puzzlingly—without consulting any dictionary, rulebook, or formal specification of what these words mean.

This is the miracle of human comprehension, and for most of human history, we took it so completely for granted that we never thought to ask how it worked. Meaning seemed self-evident, the obvious property of words that any competent speaker possessed. But the moment we try to build machines that understand—to create artificial systems capable of processing language the way humans do—we confront the riddle that lies at the heart of cognitive science: What, exactly, is meaning?

The question seems deceptively simple until you attempt to answer it rigorously. Is meaning something that exists in the world, captured like a photograph by our words? Is it a mental construction, assembled from primitive concepts stored in our neurons? Is it purely formal, the product of grammatical rules operating on arbitrary symbols? Or is it grounded in something deeper—embodied in our physical experience of a three-dimensional world, encoded somehow in the patterns of our sensory-motor systems?

For decades, artificial intelligence researchers attempted to answer this question by taking the most literally conceivable approach: they would represent meaning through symbols. Words would be tokens, definitions would be rules, and understanding would be the result of manipulating these symbols according to logical principles. This approach—what we now call symbolic AI—dominated the field for nearly forty years and produced both spectacular achievements and spectacular failures. It also left us with a profound puzzle about the nature of machine intelligence that we are only now beginning to resolve.

The resolution, surprisingly, involves geometry. Not the geometry of Euclid or Descartes that we learn in school, but something more exotic: high-dimensional spaces where meaning itself becomes a property of position and distance, where concepts are not discrete tokens but distributed patterns, and where understanding emerges from relationships rather than definitions. This is the geometry of thought, and it represents nothing less than a revolution in how we conceptualize both human cognition and artificial intelligence.

In this opening chapter, we will explore this revolution from its philosophical foundations to its practical implications. We will see why the symbolic approach failed to capture meaning, examine the cognitive science that pointed toward a geometric alternative, trace the technical developments that made distributed representations computationally tractable, and confront the mysterious effectiveness of neural language models—a phenomenon that challenges our understanding of both machines and minds. By the end, we will have established the central thesis of this book: that meaning is fundamentally geometric, that understanding arises from the relationships between concepts rather than the concepts themselves, and that this geometric perspective provides the architecture for the semantic search systems that are transforming how we interact with information.

The Symbolists and Their Grand Ambition

To appreciate the depth of the geometric turn, we must first understand what it displaced. The symbolic approach to artificial intelligence emerged in the 1950s and 1960s, drawing inspiration from several sources: the formal logic developed by philosophers like Bertrand Russell and Alfred North Whitehead, the computational theories of Alan Turing, and the structuralist tradition in linguistics exemplified by Noam Chomsky's syntax-first theories. The central commitment of this approach was elegant in its simplicity: minds, whether biological or artificial, process information by manipulating symbols according to rules.

Consider how this works in practice. In a symbolic system, the word "cat" might be represented by a token—say, the string "CAT"—in a computer's memory. The meaning of "cat" would be specified by a definition, perhaps stored as a structured list of properties: has_fur(TRUE), is_mammal(TRUE), makes_sound(purring), size(small to medium), and so on. Sentences would be parsed according to grammatical rules, producing parse trees that represent syntactic structure. And understanding would be computed by applying logical inference rules to these symbolic representations.

This approach had much to recommend it. It offered transparency: you could inspect the symbols, examine the rules, and trace the inference steps. It provided compositionality: the meaning of a complex expression was derived from the meanings of its parts through systematic rules, just as in formal logic. And it connected naturally to centuries of work in philosophy and linguistics, giving it intellectual credibility and theoretical depth.

The grand ambition of symbolic AI was nothing less than general intelligence. If you could represent all knowledge as symbols and encode all reasoning as rule application, you would have, in principle, a system capable of understanding anything and reasoning about everything. The early pioneers of AI—Marvin Minsky, John McCarthy, Allen Newell, and Herbert Simon—believed they were on the verge of cracking the code of intelligence itself. Herbert Simon was famously optimistic, predicting in 1965 that "machines will be capable, within twenty years, of doing any work a man can do."

That prediction, like many others from the early enthusiasm of AI, proved spectacularly wrong. But the symbolic approach did produce genuine achievements. Expert systems, developed in the 1970s and 1980s, demonstrated that encoding domain-specific knowledge in symbolic form could yield practical, useful systems. MYCIN, developed at Stanford in the early 1970s, used symbolic rules about bacterial infections to recommend antibiotic treatments with an accuracy that rivaled human experts. DENDRAL, another expert system from the same era, could identify the molecular structure of organic compounds from mass spectrometry data. These systems worked—not by any means of general understanding, but by applying carefully crafted rules to carefully represented knowledge.

Yet the limitations of the symbolic approach became increasingly apparent as researchers attempted to scale beyond narrow domains. The knowledge acquisition bottleneck was perhaps the most fundamental problem: encoding human knowledge into symbolic form required human experts who could articulate what they knew, knowledge engineers who could translate that knowledge into formal representations, and vast amounts of time and effort for each new domain. The system that could diagnose bacterial infections could not understand a children's story. The program that could identify chemical compounds could not answer simple questions about everyday objects.

More fundamentally, the symbolic approach struggled with the problem of meaning itself. How do symbols get their meaning? In a purely symbolic system, tokens like "CAT" are meaningful only because we, as designers and users, agree to interpret them in certain ways. The system itself has no access to cats, no experience of fur or purring, no embodied understanding of what it means to be a small furry ma