Why Robots Must Imagine: The Promise of Embodied Intelligence

When you step off a curb and see a car approaching from the left, your brain does not freeze and consult a database of images. In the fraction of a second before your foot returns to the sidewalk, your visual system has already run a small private simulation: where the car will be in 200 milliseconds, whether your weight is shifting forward, whether the asphalt is slick, whether the driver has seen you. You have imagined a future that does not yet exist, and you have used that imagined future to choose an action. This is not magic; it is the everyday miracle of having a body that has spent years learning how the world behaves.

A modern large language model trained on essentially the entire public internet cannot do this. Ask it to describe what happens when you drop a glass, and it will give you a fluent paragraph about shards on a kitchen floor. Ask it to predict, with any quantitative accuracy, how the glass will rotate, where the shards will land, or how the sound will propagate through the room, and it will fail. The model has read about breaking glass ten thousand times, but it has never felt a glass leave its fingers, never watched a coffee table rotate under its fall, never flinched at the sound. Its competence is real, but it is the competence of a brilliant, well-read observer standing outside the window. When the window opens and the physical world rushes in, that observer has nothing to push against.

The argument running through this book is that any machine that wishes to act competently in the physical world must possess, in some form, an internal model of that world. The claim is not new. Engineers have been building such models since the 1950s, and psychologists have been studying the biological version since at least the time of Helmholtz in the nineteenth century. What is new, and what makes this a particularly urgent moment for the field, is that the latest wave of artificial intelligence has produced systems that are spectacularly good at the symbolic, disembodied parts of intelligence and spectacularly bad at the physical, embodied parts. The result is a strange and instructive moment in which a chatbot can pass a law school exam but a robot cannot reliably pour water into a glass. Understanding why this gap exists, and how to close it, is the project of the fifteen chapters that follow.

The opening argument moves from problem to history to structure. It begins by explaining the specific ways in which disembodied models fail when faced with physical causality. It then traces the intellectual lineage from cybernetics to modern deep reinforcement learning, showing that the current excitement around "world models" is in fact the third time the field has rediscovered the same idea. It introduces the three functional pillars that any embodied intelligence must implement — perception, prediction, and action — and shows how they form a closed loop. It grounds these abstractions in concrete motivating scenarios: a self-driving car facing a child chasing a ball, a warehouse robot asked to pick a single strawberry from a cluttered bin, an exoskeleton helping an elderly climber cross a patch of ice. It then previews the structure of the book and gives the reader a hands-on thought experiment that will serve as a recurring point of reference throughout.

The Disembodied Ceiling: Why Pure Models Hit a Wall

To understand why robots need world models, it helps to begin by examining why machines that lack them are so impressive in some ways and so brittle in others. The class of models in question has many names — foundation models, large language models, multimodal models, generative pretrained transformers — but for the purposes of this book we will refer to them collectively as text-and-image models, because their training data and their representational substrate are dominated by text and still images.

The strengths of these models are well documented and will not be belabored here. They can summarize documents, translate between languages, write code that often runs, explain jokes, identify objects in photographs with superhuman accuracy, and engage in dialogue that is indistinguishable from a thoughtful human interlocutor for several turns at a time. What matters for our purposes is what they cannot do, and the list of things they cannot do is structured rather than incidental. It traces back to the specific way these systems are trained.

The first limitation is missing physical ground truth. Text is a record of what humans have chosen to write down about the world, not a record of the world itself. When a physicist writes "the ball follows a parabolic trajectory under uniform gravity," the model learns a sentence, not a parabola. It has access to the *description* of the law but not to the dense, continuous, noisy stream of sensory experience that a child accumulates while watching balls fall, throwing balls herself, and being hit by balls unexpectedly. The description is highly compressed; the experience is not. A model that has only descriptions must learn to interpolate within the space of sentences, not within the space of physical states. When the physics diverges from the description — because the ball is wet and aerodynamic drag matters, because the room has noticeable air resistance, because the ball is spinning — the model has no way to know, because the training corpus never recorded the divergence.

The second limitation is the absence of action and consequence. A text-and-image model never performs an action and observes its outcome. It has no proprioception, no sense of effort, no surprise when a stack of blocks unexpectedly topples. Its training distribution contains the word "surprise" and many sentences that describe what surprises feel like, but the system itself has never been surprised. Action is what closes the loop between prediction and reality. Without action, prediction is merely pattern-matching on what others have said will happen. With action, prediction becomes testable, and the discrepancy between prediction and outcome becomes the primary signal for learning. This is the difference between reading a thousand recipes and actually cooking: the cook encounters the gap between predicted and actual outcomes on every dish, and learns from each gap.

The third limitation is the missing sense of time. Text and images are, in their canonical form, static. A paragraph is read in sequence; an image is viewed as a whole. Neither presents the world as the continuous, four-dimensional unfolding that an embodied agent experiences. Even when models are extended to handle video, the temporal horizon is typically short and the training objective is usually reconstruction or caption-matching, not the prediction of physical consequences tens of seconds into the future. A robot that cannot imagine itself five seconds from now cannot plan its grip on a fragile object, cannot anticipate a pedestrian's trajectory, cannot recover from a stumble. The temporal density of lived experience is something the static corpus cannot match.

A useful way to make this concrete is to compare two systems attempting the same simple task: predicting whether a stack of three wooden blocks, placed on a slightly wobbly table, will topple when a small lateral nudge is applied to the bottom block. A human child of perhaps four years old, after watching this happen a handful of times, will look at the alignment of the blocks, the friction of the surface, the weight distribution, and produce a confident yes-or-no in under a second, along with a short causal explanation ("because the bottom one is sticking out too far"). A state-of-the-art vision-language model given a photograph of the setup and asked the same question will, with comparable confidence, produce an answer that is essentially a coin flip, accompanied by plausible-sounding language that has no causal backing. The child has a generative model of stacked-block dynamics,