Foundations of World Models and Embodied Intelligence

Foundations of World Models and Embodied Intelligence

An agent walks into a kitchen, picks up a mug, and pours coffee without spilling. Why is this trivial for a toddler and nearly impossible for the most advanced humanoid robot in 2024? The toddler carries, somewhere in her skull, an implicit model of how liquids slosh, how hands grip, and what happens if the mug tilts past forty degrees. The robot has fifteen million lines of control code, but no such model — only a cascade of feedforward classifiers trained to map pixels to torques. The gap between the two is not a gap in compute, sensors, or even data. It is a gap in *predictive representation*. The toddler can imagine the consequences of her actions before she takes them; the robot can only react to what already happened. This book is about closing that gap by giving machines what Tolman, in 1948, called a "cognitive map": an internal, learnable, generative model of the world that supports planning, generalization, and credit assignment across long time horizons. We call such a model a world model, and the central thesis of this book is that world models are not an optional add-on to embodied intelligence — they are its core.

This opening chapter does four things. First, it makes precise *why* purely reactive policies fail, by isolating the four structural problems that any embodied agent must solve: partial observability, stochastic dynamics, action-conditioned prediction, and long-horizon credit assignment. Second, it introduces the perception–cognition–action loop as the organizing abstraction of the field, and shows where world models plug into that loop. Third, it gives a working taxonomy of the three architectural patterns that recur throughout modern research: explicit simulation, latent dynamics, and predictive representation learning. Fourth, it traces the historical lineage of the idea from Tolman's rats through Wiener's cybernetics, Friston's predictive coding, Ha & Schmidhuber's neural world models, and LeCun's JEPA agenda. We close with a complete, runnable example that pits a model-free Q-learner against a model-based planner in a small grid world, and we preview the mathematical formalism that Chapter 2 will introduce. The reader who finishes this chapter should be able to (a) define "world model" without hand-waving, (b) name at least three historical antecedents, and (c) explain, with a concrete experiment, why model-based agents can be more sample-efficient than their model-free cousins.

---

1. The Case for Internal Models: Why Reactive Policies Are Not Enough

To see why a policy that maps observations directly to actions — the canonical approach of end-to-end behavior cloning and of the early deep reinforcement learning literature — is structurally limited, we have to look at the problem rather than the solution. An embodied agent does not get to see the world the way a chess program sees a board. It receives a stream of high-dimensional, noisy, *partial* observations; it acts on an environment whose dynamics may be stochastic and whose state may be far larger than any single observation; and it must learn, from sparse and delayed reward signals, which of its trillions of past decisions actually mattered. These four properties — partial observability, stochastic dynamics, action-conditioned prediction, and credit assignment — together define the hard core of embodied intelligence, and a reactive policy is one that simply refuses to model any of them.

Partial observability is the most obvious obstacle. A robot arm equipped with a wrist camera does not see the joint torques, the masses of the objects on the table, or whether the drawer behind it is open. Formally, the agent does not have access to the Markov state $s_t \in \mathcal{S}$; it only has access to an observation $o_t \in \mathcal{O}$ drawn from an emission distribution $o_t \sim E(\cdot \mid s_t)$. A reactive policy $\pi_\theta(a_t \mid o_t)$ is forced to act on $o_t$ as if it were $s_t$. If the relevant information for choosing the right action is not in $o_t$, no amount of policy capacity will recover it. A world model, by contrast, can maintain a *belief state* $b_t$, an internal summary of everything the agent has seen, and act on the belief rather than the raw observation. The belief is constructed by the model, so the model is what makes the policy work. A canonical formalization is the partially observable Markov decision process (POMDP), and the standard result is that optimal policies in POMDPs generally need memory — that is, internal state — and that internal state is most usefully interpreted as a sufficient statistic of observation history. Sufficient statistics are a kind of world model.

Stochastic dynamics compound the problem. Even if the agent had full state access, the transition kernel $P(s_{t+1} \mid s_t, a_t)$ might be noisy: the same action from the same state can lead to different successors. Consider a quadruped walking on gravel. Each footfall has a slightly different friction coefficient. A reactive policy that has memorized the mean outcome will be wrong on the variance. A world model can either (a) predict the distribution over successors and plan under uncertainty, or (b) at minimum estimate the variance and avoid actions whose consequences are high-variance. The same point applies to partial observability with a vengeance: if the emission is stochastic, no amount of memory of past observations can fully recover the state, and the agent must plan in the space of distributions. We will return to this in Chapter 4 when we discuss distributional and probabilistic world models.

Action-conditioned prediction is the third structural problem, and the one most often overlooked by people coming from supervised learning. In supervised learning, we have pairs $(x, y)$ and we want to learn $p(y \mid x)$. In embodied intelligence, we want to learn $p(s_{t+1}, r_{t+1} \mid s_t, a_t)$ — but the *actions* are not i.i.d. samples from a fixed distribution. They are chosen by the agent itself, in order to elicit information. The act of choosing an action changes what the next training example will look like. This is the exploration problem, and it is mathematically distinct from supervised learning: the agent's uncertainty about the model is coupled to its uncertainty about which data it will collect next. Reactive policies that do not maintain an explicit model of consequences are forced to use heuristics like $\epsilon$-greedy or upper confidence bounds, which are essentially *implicit* models of action-conditioned uncertainty. A world model makes this explicit: the agent can ask, for any candidate action sequence, "what does my model predict will happen, and how confident am I?" and choose accordingly.

Credit assignment is the fourth, and in many ways the deepest, problem. A long-horizon task — say, a robot clearing a dinner table — consists of perhaps a thousand motor commands, and the only reward signal is delivered at the end, when the table is empty. The gradient of that final reward with respect to any one of the thousand actions is, in general, either zero (if the action's effect is washed out by stochasticity) or astronomically small (if the effect is real but buried under noise). This is the credit assignment problem, and it is the reason that vanilla policy gradient methods require millions of episodes to learn even moderately complex behaviors. A world model helps in two distinct ways. First, it can be used to *imagine* rollouts — generate synthetic trajectories of what *would* happen under candidate policies, evaluate them, and update the policy on imagined experience. This is the Dyna trick of Sutton (1991), and it is the conceptual ancestor of every modern model-based RL system. Second, a world model that is itself learned by gradient descent can be *differentiated through*: the policy can be updated by backpropagating through the model'