Embodied Intelligence and the Necessity of World Models

An agent embedded in a physical or simulated environment must constantly negotiate a tradeoff between two epistemic commitments: it can either absorb the statistics of the world directly into a reactive policy that maps observations to actions, or it can construct an internal generative model of how the world unfolds and use that model to deliberate, plan, and imagine. The first commitment is parsimonious but myopic; the second is demanding but elastic. This chapter argues that, under conditions that are common rather than exceptional in embodied intelligence, the second commitment dominates the first, and we formalize what "constructing an internal model" precisely means.

The case for the world-model paradigm rests on three pillars. First, a deep cognitive lineage: from Helmholtz's doctrine of unconscious inference, through Craik's mental models, Tolman's latent learning, to the modern apparatus of model-based reinforcement learning. Second, a clean formal substrate: the partially observable Markov decision process, in which an encoder, a dynamics function, a reward or cost predictor, and a policy can be carved out as distinct functional modules with provable roles in optimization. Third, an empirical and theoretical sample-complexity gap: when the dynamics are learnable and the planning horizon is nontrivial, agents that exploit a learned model converge faster and generalize farther than those that do not. The remainder of this chapter develops all three pillars, closes with a reproducible reference implementation, and prepares the reader for the mathematical machinery of Chapter 2.

A Cognitive and Computational Genealogy of the World Model

The notion that an intelligent system requires an internal surrogate of reality predates the existence of computers by nearly a century. In 1867, Hermann von Helmholtz, working on perceptual physiology, proposed that vision is not a direct readout of the retinal image but a process of *unconscious inference* in which the brain selects the most probable world state consistent with sensory evidence and prior expectations. The operation is generative in form: the agent maintains a hypothesis about hidden causes, updates that hypothesis when evidence arrives, and uses it to predict forthcoming observations. Helmholtz's framework introduced two ideas that remain central to contemporary world models: that perception is fundamentally an inference problem over latent variables, and that prediction is its principal currency.

The British psychologist Kenneth Craik extended this line of argument in 1943 with a striking claim in *The Nature of Explanation*: human reasoning operates by means of small-scale models of external reality and of its own possible actions, models that allow it to try out alternatives, to react to events long after they have occurred, and to weigh the merits of different courses of action without committing to any of them. Craik's "mental model" was a structural analog: a token process whose behavior parallels that of the referent process so that conclusions drawn from the model transfer to the world. This is the conceptual ancestor of every learned dynamics function we will encounter, from tabular transition matrices in gridworlds to deep latent dynamics in simulated robotics. The crucial move Craik made was to treat the model not as a luxury but as the principal instrument of intelligence. A policy without a model, he argued, is forced to discover consequences by suffering them.

Edward Tolman's 1948 latent-learning experiments supplied the first behavioral evidence. Rats that wandered a maze without reward later solved it dramatically faster than rats introduced to the maze for the first time, provided they had been allowed to explore it earlier. The implication was that the rats had constructed a *cognitive map* of the maze during the unrewarded phase and reused it when reward became available. Pure stimulus-response theories of behavior, which correspond in modern terms to model-free policies, had no vocabulary to capture this transfer. Tolman's cognitive map is a world model. The paradigm shift he inaugurated, that learning can occur without immediate reinforcement and be cashed in later, is the conceptual foundation on which Sutton's Dyna architecture would later rest.

The translation into computational form took several decades. Samuel's checkers program (1959) used lookahead search over a learned evaluation function, an early hybrid of model-based search and learned value estimation. In the 1980s, the formal apparatus of Markov and partially observable Markov decision processes gave the model-based paradigm a rigorous spine. Werbos, Watkins, and others introduced the temporal-difference and Q-learning machinery that eventually matured into the model-free family. The reunification came with Sutton's 1991 *Dyna* architecture, which interleaved direct reinforcement learning with planning over a learned world model and demonstrated order-of-magnitude sample-efficiency gains on gridworld tasks. From that point onward, the world-model paradigm became a recurring research program, renewed in the 2000s by PILCO, in the 2010s by guided policy search and DeepMind's AlphaZero-style planners, and in the late 2010s and 2020s by Dreamer, DreamerV2, DreamerV3, GAIA-1, UniSim, and a torrent of large-scale latent dynamics models.

It is worth pausing to name what the lineage commits us to. The Helmholtz tradition commits us to inference: world models are not lookup tables but probabilistic surrogates that combine prior and likelihood. The Craik tradition commits us to functional isomorphism: a world model is useful insofar as rolling it forward yields predictions that transfer to the world. The Tolman tradition commits us to deferred use: the value of a model is realized not at the moment of construction but at the moment of deployment, often in regimes the agent never visited during learning. Modern model-based reinforcement learning is the conjunction of these three commitments, instantiated in a learnable function approximator and a planning or policy-optimization routine that exploits it.

Formalizing the World Model in the POMDP Substrate

The cleanest formal home for the world-model paradigm is the partially observable Markov decision process. A POMDP is a tuple $$ \mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{O}, T, R, O, \gamma, \rho_0) $$ where $\mathcal{S}$ is the unobservable state space, $\mathcal{A}$ the action space, $\mathcal{O}$ the observation space, $T(s' \mid s, a)$ the transition kernel, $R(s, a)$ the reward function, $O(o \mid s)$ the emission kernel, $\gamma \in [0, 1)$ the discount factor, and $\rho_0$ the initial state distribution. The agent at time $t$ receives an observation $o_t \sim O(\cdot \mid s_t)$, executes an action $a_t \sim \pi(\cdot \mid h_t)$ where $h_t = (o_{\le t}, a_{<t})$ is the interaction history, transitions to $s_{t+1} \sim T(\cdot \mid s_t, a_t)$, and receives $r_t = R(s_t, a_t)$. The objective is to maximize the expected discounted return $J(\pi) = \mathbb{E}_\pi[\sum_{t=0}^{\infty} \gamma^t r_t]$.

A *world model* in the sense used throughout this book is a learnable, factored approximation to the generative structure of $\mathcal{M}$ restricted to the quantities an agent needs to act. Canonical decompositions take three forms. In the *observation-space* form, the model predicts the next observation $\hat{o}_{t+1}$ and the reward $\hat{r}_{t+1}$ directly from $o_t$ and $a_t$. In the *latent-space* form, an encoder first compresses history into a stochastic latent state $z_t$, and the dynamics are then expressed over $z_t$. In the *state-space* form, when privileged state information is available, the model predicts $s_{t+1}$ directly. The latent-space form is the most common in modern deep agents because it lets the model learn a representation that is simultaneously compact and predictive. We adopt it as our reference.

Concretely, we posit t