World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applications

TL;DR

An agent that only reacts must try things in the real world to find out what happens. An agent with a world model can try things inside its own head first. That is the whole idea: learn a compressed internal state of the environment, learn how that state changes under each action, then plan, train, or reason inside the learned simulation instead of paying real-world cost.

The field grew fast and fragmented. Reinforcement-learning people build small latent dynamics models (PlaNet, Dreamer, MuZero). Video people build billion-parameter generators (Sora, Genie, Cosmos). Neither group evaluates the same way, and the word “world model” now means both. This survey imposes a four-axis taxonomy — architecture, method family, reasoning strategy, application domain — over 400+ papers and shows where the axes actually interact.

The honest headline is a negative one. On WorldBench, every state-of-the-art visual world model scores about 45% foreground mIoU on physical-reasoning tasks and degrades sharply after 5–9 predicted frames. Models that look photorealistic still break object permanence and conservation laws. The survey’s most useful contribution is not the taxonomy; it is the argument that the field’s dominant metric (FVD) rewards pretty frames over correct physics, and that this has been quietly steering research the wrong way for years.

Problem & Motivation

The pain in one sentence: every real decision an agent makes costs time, money, or safety, and today’s models have no reliable way to find out what would happen before committing.

Large language models are the sharpest example. GPT-4-class systems are excellent inside the discrete, low-dimensional space of text tokens and bad at three things a housecat does effortlessly: holding a persistent state of the physical world, reasoning about cause, and planning over a long horizon. The survey frames this as Moravec’s paradox — chess and essays turned out to be the easy part, sensorimotor competence the hard part.

Model-free reinforcement learning has the complementary problem. It learns which action pays without ever learning how the world works, so it needs enormous interaction budgets, transfers badly when the reward changes, and cannot answer “what if I had turned left instead?” because it has no simulator to re-run.

The concrete costs that motivate the field:

• Interaction is expensive. A robot arm learning by trial and error breaks hardware. A driving policy learning by trial and error kills people. A trading policy learning by trial and error loses money that does not come back.
• Rare events dominate risk, and you cannot collect them. The tail — freak weather, an atypical patient, an unusual road geometry — is exactly where prediction matters most and data is thinnest.
• Reactive policies do not compose. Retrain the reward, throw away everything the agent knew about the environment.

The prior fix — hand-built simulators (CARLA, MuJoCo, physics engines) — has its own ceiling. Simulators encode a human’s model of the world, complete with rigid-body contact assumptions and Coulomb friction that diverge from real mechanics precisely in the contact-rich tasks robots most need. So the field moved to learning the simulator from data. That is a world model.

Where prior surveys fell short, per the authors: Ding et al. focus on understanding-vs-prediction; Li et al. give a three-axis taxonomy for embodied AI only; other surveys cover driving, manipulation, or 3D scenes in isolation. None cover architecture, method, reasoning, and application together — which matters because the interesting findings are cross-axis (e.g. pixel-space diffusion beats token-space transformers on Atari specifically because small visual details like ball position survive).

What’s New (Core Contribution)

This is a survey, so “novel” means organizational and critical, not algorithmic. Four things are genuinely new; one is repackaged.

1. A four-axis taxonomy that separates independent design choices. Before: world models were listed by lineage (“the Dreamer family”, “the diffusion ones”) or by application (“driving world models”). Now: four orthogonal axes — architecture (how state is represented, how dynamics are formulated, which modalities, which learning paradigm), method family (RSSM / transformer / diffusion / physics-informed / language-augmented), reasoning strategy (imagination planning / policy learning / counterfactual / planning under uncertainty), application domain. The payoff is that you can now say “GAIA-1 is discrete-token + transformer + generative-simulator + driving” and “MuZero is task-aligned-latent + recurrent + forward-search + games”, and see that these differ on three axes, not one.

2. A definitional split the field has been avoiding. The authors argue for separating predictive world models (action-conditioned, support decision-making) from generative world simulators (produce realistic sensory data, may have no action or reward at all). Sora is the second and gets called the first. This is not pedantry: it is why a video model with a great FVD and a control model with a great human-normalized score cannot be compared, and why claims about “emergent physics understanding” keep collapsing under scrutiny.

3. A sustained critique of the field’s own metrics. FVD — the de facto standard — is shown to be biased toward per-frame appearance over temporal realism. You can halve FVD by choosing static, motion-free videos. Ge et al. and Luo et al. document non-Gaussian feature distributions, insensitivity to temporal distortion, and impractical sample-size requirements. Diagnostic replacements (WorldBench, Physion++, IntPhys, VBench 2.0, JEDi) are catalogued, and their verdict is consistent: current models perform near chance on violation-of-expectation tests.

4. Extension into domains nobody had surveyed. Medical imaging (disease-progression latents, tumor evolution, surgical video), educational measurement (the “environment” is a human learner whose state is only indirectly observable), and business/finance (where deploying the model changes the environment it models — reflexivity, which breaks the standard POMDP framing outright).

Repackaged, not new: the historical narrative (Minsky’s frames → Johnson-Laird’s mental models → Ha & Schmidhuber → LeCun’s JEPA → foundation simulators) appears in most world-model surveys. Treat it as context, not contribution.

How It Works (Technically): The Taxonomy

For a survey, the mechanism is the taxonomy. Below: the shared anatomy every world model has, then the four axes, then a comparison table, then three mini deep-dives on the techniques worth actually building.

The shared anatomy

Nearly every system in the survey decomposes the same way. Formally the model approximates

$$p_\theta(s_{t+1}, o_{t+1}, r_t \mid s_t, a_t)$$

In plain English: given where I think I am and what I do next, give me a distribution over where I end up, what I will see, and what reward I get. The word “distribution” is load-bearing — a point estimate cannot represent “the pedestrian goes left or right”.

Four modules implement it:

The decoder being optional is one of the survey’s most useful teaching points. MuZero has none: its latent is trained only to predict reward, value, and policy, and it still reached superhuman Go/chess/shogi/Atari. JEPA has none either. If your latent is optimized to reconstruct pixels, you spend capacity on texture; if it is optimized to predict decision-relevant quantities, you spend capacity on what matters.

Three properties separate a world model from a generic predictor: it is action-conditioned (supports “what if I turn left”), it rolls out multi-step (apply it to its own output, get a trajectory), and its predictions are used downstream (planning, policy training, safety checks) rather than being the end product.

Architecture & data flow

flowchart LR
  O[Observation o_t] --> ENC[Encoder q_phi]
  A[Action a_t] --> DYN
  ENC -->|latent z_t| DYN[Dynamics p_theta]
  DYN -->|z_t+1| HEADS[Reward + continue heads]
  DYN -->|z_t+1| DEC[Decoder - optional]
  HEADS -->|r_hat, gamma_hat| CTRL[Controller: policy, search, or planner]
  DEC -->|o_hat| VIS[Human inspection]
  CTRL -->|next action| DYN
  CTRL ==>|chosen action| ENV[(Real environment)]
  ENV -.->|new observation| O

Read the thick arrow as the only place real-world cost is paid. Everything on the loop between DYN and CTRL is free imagination.

The four axes

flowchart TD
  WM[World model design space]
  WM --> AX1[Axis 1: Architecture]
  WM --> AX2[Axis 2: Method family]
  WM --> AX3[Axis 3: Reasoning strategy]
  WM --> AX4[Axis 4: Application domain]

  AX1 --> R1[Representation: pixels / continuous latent / discrete tokens / joint-embedding / object slots / 3D occupancy]
  AX1 --> R2[Dynamics: deterministic / stochastic / implicit generative / representation-space / memory-augmented]
  AX1 --> R3[Modality: vision / language / 3D / tactile / fused]
  AX1 --> R4[Learning: self-supervised / online MBRL / offline / foundation pretrain / imitation / hybrid]

  AX2 --> M1[RSSM and recurrent]
  AX2 --> M2[Transformer]
  AX2 --> M3[Diffusion]
  AX2 --> M4[Physics-informed]
  AX2 --> M5[Language-augmented]

  AX3 --> S1[Imagination planning: background vs decision-time]
  AX3 --> S2[Policy learning in imagination]
  AX3 --> S3[Counterfactual reasoning]
  AX3 --> S4[Planning under uncertainty]

  AX4 --> D1[Robotics / driving / video / agents / RL / science / medicine / education / finance]

Axis 1 — Architecture. The representation sub-axis is the consequential one. Six families, with a real trade-off in each:

• Pixel-space (DIAMOND, GameNGen): nothing is discarded, but a 256×256 frame is 196,608 dimensions and most of them are texture. Correct choice only when visual fidelity is the product.
• Continuous latent (World Models, PlaNet, Dreamer): the workhorse. RSSM splits state into a deterministic recurrent part h_t that carries memory and a stochastic part z_t that carries uncertainty. Blurry reconstructions are the known cost of the Gaussian likelihood.
• Discrete tokens (IRIS, DreamerV2, GAIA-1): quantize to a VQ-VAE vocabulary, then dynamics is literally next-token prediction — you get to reuse the entire LLM toolchain. DreamerV2 found 32 categoricals × 32 classes beats Gaussian latents on Atari, probably because game state transitions are themselves discrete. Cost: quantization throws away fine spatial detail.
• Joint-embedding / JEPA (I-JEPA, V-JEPA 2): predict the embedding of the next observation, never the observation. No decoder at all.
• Object slots (C-SWM, RoboDreamer): decompose state into per-entity slots, model interactions with a graph net. Buys combinatorial generalization and interpretability; costs O(N²) in object count and assumes the scene decomposes cleanly.
• 3D / occupancy (OccWorld, Copilot4D): voxels and point clouds. Essential for driving; memory scales cubically with resolution.

Axis 2 — Method family. How the dynamics function is parameterized. This is where the one strong comparison table belongs (below).

Axis 3 — Reasoning strategy. How predictions become decisions. The survey’s cleanest distinction is when the optimization happens:

• Background planning (Dreamer, PlaNet): imagine during training, amortize the result into a policy’s weights, act fast at inference.
• Decision-time forward search (MuZero, EfficientZero, TD-MPC): imagine at inference, run MCTS or trajectory optimization fresh from the current state. Costs compute per decision; buys the ability to recover from out-of-distribution states the policy never saw.
• Counterfactual (abduction → action → prediction): re-run history under a different decision, holding everything else fixed.
• Planning under uncertainty: split aleatoric (the world is genuinely random) from epistemic (I have not seen this region) and handle each differently.

Axis 4 — Application domain. Robotics, driving, video, multimodal agents, RL/games, science, medicine, education, finance. The survey’s own observation: the paradigm is flexible enough to reach all of them, and each one breaks a different assumption.

Schematic 3D map of the design space: representation family (x), how the dynamics are parameterized (y), and how the model is used (z). Each sphere is a named system from the survey; drag to orbit. Note that the well-known systems are spread far apart — they are not competitors on one axis, they are different points in a four-dimensional space.

One strong comparison table: the five method families

The single most useful sentence in the table: no family dominates. RSSM is cheapest, transformers remember longest, diffusion looks best, physics-informed generalizes furthest on the narrow problems it fits, and language gives you compositional task specification. Pick by deployment constraint, not by fashion.

Mini deep-dive 1: latent imagination (the Dreamer loop)

This is the mechanism to understand first, because everything else is a variation on it.

Given a candidate action sequence a_{t:t+H}, roll the latent forward:

$$z_{t+k} \sim p_\theta(z_{t+k} \mid z_{t+k-1},, a_{t+k-1}), \quad k = 1 \dots H$$

Then score each imagined state with the reward head and value function:

$$r_{t+k} \sim p_\phi(r \mid z_{t+k}), \qquad v_{t+k} = \mathbb{E}[V(z_{t+k})]$$

and optimize expected discounted return inside the model:

$$J(\pi) = \mathbb{E}{p\theta,\pi}\left[\sum_{k=0}^{H-1} \gamma^k r_{t+k}\right]$$

What this actually does: it converts future interaction from something you have to do into something you can differentiate. Because the transition network is a differentiable neural net, you can backpropagate a long-horizon value estimate through the entire imagined trajectory in one pass. That is the trick. Multi-step credit assignment, which model-free RL solves with high-variance policy gradients, becomes ordinary backpropagation through time.

Three consequences the survey draws out:

• Data amplification. One real trajectory expands into thousands of imagined ones. PlaNet’s 50× data-efficiency claim comes from here.
• Risk-free exploration. You can crash the car ten thousand times in latent space.
• Free counterfactuals. Fix z_t, vary the action, and you have an apples-to-apples “what if” — no confounding, because the starting state is literally identical.

And the failure mode, which is the same mechanism running backwards. Janner et al. proved the return discrepancy between model and reality grows linearly in both the rollout length k and the single-step error ε. A 1% per-step error diverges catastrophically within tens of steps. Worse, the policy actively seeks the divergence: it finds trajectories that score brilliantly under the model and fail in reality. This is model exploitation, and it is the field’s central unsolved problem.

Schematic of one planning step. From the current latent state the model branches over candidate actions, rolls each forward, and scores the leaves with the reward predictor and value head. The widening blue haze is the growing gap between imagined and real trajectories — this is compounding error, and it is why almost every practical system truncates the horizon.

Mini deep-dive 2: counterfactual reasoning (abduction → action → prediction)

The most commercially interesting technique in the survey, and the least implemented.

Standard prediction answers “what happens next?”. Counterfactual reasoning answers “given what actually happened, what would have happened if I had decided differently?” — Pearl’s top rung. Three steps:

$$z_t \sim q_\phi(z_t \mid o_{\le t}, a_{<t}), \qquad a_t \leftarrow do(a’t), \qquad z’{t+1} = f_\theta(z_t, a’_t)$$

1. Abduction. Infer not just the latent state but the exogenous noise — the specific unobserved randomness that produced this particular observed trajectory. This is the hard step and it is underdetermined: many latent configurations explain the same observations.
2. Action. Apply the do operator: swap the factual action for the counterfactual one and change nothing else. Every other environmental factor stays rigidly fixed.
3. Prediction. Roll forward from the same inferred state under the new action.

Why the fixed state matters so much: factual and counterfactual trajectories share the identical inferred z_t. That is what isolates the causal effect of the decision from the environment’s own behavior. Naive re-simulation from a nearby state does not do this and is not a counterfactual.

Where it pays: offline RL (re-interpret a logged dataset as a causal substrate without new interaction), incident review in driving and medicine (“was this collision preventable?”), and decision auditing generally.

The two limits, stated bluntly by the authors:

• Non-identifiability. Multiple distinct latent dynamics models can fit the observed data perfectly and still give completely different counterfactual answers. Observational accuracy does not imply counterfactual validity — ever.
• Counterfactual hallucination. A counterfactual action far from the data-collecting policy pushes the model out of distribution. The rollout stays internally coherent and becomes causally false. It looks right and is wrong, which is the worst kind of wrong.

The fix direction: structural causal models baked into the architecture, enforcing that intervening on one mechanism produces localized latent changes rather than perturbing a monolithic vector.

sequenceDiagram
  participant H as Observed history
  participant E as Encoder / abduction
  participant D as Dynamics f_theta
  participant C as Comparator
  H->>E: o<=t, a<t
  E->>D: latent z_t + inferred exogenous noise
  Note over D: factual branch keeps a_t
  D->>C: factual outcome
  Note over D: counterfactual branch does do(a'_t), same z_t
  D->>C: counterfactual outcome
  C-->>H: causal effect of the decision, environment held fixed

Mini deep-dive 3: the LLM as the transition function (RAP)

The technique that transfers most directly into an existing agent stack, because it needs no training.

Reasoning via Planning (RAP) casts one LLM in two roles. As the agent, it proposes candidate actions. As the world model, it predicts the resulting state. Monte Carlo Tree Search then does what MCTS always does — expands promising branches, backs values up the tree — except the “environment” it queries is the same LLM playing its second role.

Result: LLaMA-33B under RAP beat GPT-4 with plain chain-of-thought by 33% on plan generation. A much smaller model beat a much larger one because structured search over an explicit world model beats sequential token-by-token reasoning.

The honest caveat the survey attaches: Wang et al.’s ByteSized32-State-Prediction benchmark shows current LLMs are unreliable state simulators. They carry rich prior knowledge and lack causal fidelity for multi-step simulation. RAP works when the state is short, symbolic, and checkable; it degrades when it is long, continuous, or physical.

The algorithm, simplified

The core loop that unifies the whole survey — learn dynamics from real data, then improve the policy entirely inside imagination.

# Stubs: encode(o) -> latent; step(z, a) -> next latent; reward(z), value(z) -> scalars
# Everything below is differentiable; that is the whole point.

HORIZON = 15          # DreamerV3's actual imagination horizon
GAMMA   = 0.99

def train_world_model(real_batch):
    """Fit dynamics on REAL experience only. This is the only place reality is used."""
    loss = 0.0
    z = encode(real_batch.obs[0])
    for t in range(len(real_batch.obs) - 1):
        z_pred = step(z, real_batch.actions[t])              # prior: where the model thinks it goes
        z_post = encode(real_batch.obs[t + 1])               # posterior: where it actually went
        loss += kl(z_post, z_pred)                           # dynamics must match reality...
        loss += mse(reward(z_pred), real_batch.rewards[t])   # ...and so must the reward head
        z = z_post                                           # teacher-forcing: feed the truth back
    return loss

def improve_policy_in_imagination(start_states, policy):
    """Zero real interaction happens here. Every step is a hallucination we can differentiate."""
    returns = []
    for z in start_states:                    # seeded from REAL states in the replay buffer
        traj_return, discount = 0.0, 1.0
        for k in range(HORIZON):              # short on purpose: error compounds ~linearly in k
            a = policy(z)                     # actor acts inside the dream
            z = step(z, a)                    # dynamics steps the dream forward
            traj_return += discount * reward(z)
            discount *= GAMMA
        traj_return += discount * value(z)    # bootstrap past the truncated horizon
        returns.append(traj_return)

    # The key line: gradients flow back through EVERY imagined step into the policy weights.
    # No policy-gradient variance, no real-world rollouts. This is why it is sample-efficient.
    return -mean(returns)                     # maximize return == minimize its negative

Two lines carry the whole idea. z = step(z, a) inside the loop is imagination — the model consuming its own output. And returning a differentiable -mean(returns) is what turns “plan for the future” into “run backprop”.

Schematic of the compounding-error bound (Janner et al.): the gap between what the model promises and what reality delivers grows roughly linearly with rollout length k and with per-step error. Drag the slider to change per-step error. The shaded band is the horizon most systems actually use — note how small it has to be.

Built on Prior Work: A Map of the Field

Lineage matters here because the field keeps rediscovering the same three ideas at larger scale.

The historical arc in one line: symbols → learned latents → latents optimized for decisions → foundation-scale simulators, with the fourth stage still unproven on the physics it claims to have learned.

Results & Evidence

A survey has no experiments of its own, so what matters is what it establishes by aggregation — and what it does not.

Numbers worth carrying around:

What the evidence establishes: that learned latent dynamics genuinely improve sample efficiency over model-free RL on standard control benchmarks; that representation choice has large, measurable effects (DreamerV2’s discrete latents on Atari, DIAMOND’s pixel-space detail retention); and that short-horizon rollouts, uncertainty penalties, and ensembles are effective mitigations for model exploitation.

What it does not establish — read this part carefully:

• Physical understanding. IntPhys reports current models near chance on violation-of-expectation tests. Physion++ finds all current models fail human-level prediction when latent property inference is required. Kang et al.’s controlled experiments show video models exhibit case-based generalization — mimicking the nearest training example rather than abstracting a rule. Photorealism is not physics.
• Cross-family comparability. An RL world model measured by HNS on Atari, a driving model measured by FVD on nuScenes, and a robotics model measured by manipulation success rate cannot be meaningfully ranked against each other. Nothing in the survey fixes this.
• That reported open-loop wins survive deployment. Li et al. showed driving models get strong open-loop nuScenes scores by reading their own ego-velocity and heading, largely ignoring perception. Bench2Drive argues L2 trajectory error does not predict closed-loop driving ability at all. Any world model evaluated only open-loop looks better than it is.
• Statistical reliability. Agarwal et al. (NeurIPS 2021 Outstanding Paper) showed mean HNS is dominated by outlier games and median discards too much; they proposed IQM and stratified bootstrap CIs. The survey notes that many world-model papers still report bare mean/median with no confidence intervals. Treat single-number comparisons in this literature with suspicion.

Caveat on the survey itself: it is a broad, 400+ reference synthesis assembled by 26 authors across 12 institutions, and section quality varies. Section 3.2 (dynamics) is written almost entirely through a medical lens, which reads as an authorial interest rather than a property of the taxonomy. Several passages are duplicated near-verbatim (the “motivation for latent prediction” paragraph appears twice). Some cited systems are very recent preprints with no independent replication. Use it as a map, verify the individual claims at the source.

How You’d Use It

Mapping this to your own agent harness and automations. Three levels, increasing effort.

1. Your harness — steal the vocabulary, immediately, zero build cost. Multi-agent systems already have world models; they just are not named. Your shared state store is the encoder output. Your “what happens if this agent runs this tool” is the dynamics model. Naming it exposes the missing piece: almost no production agent system has a reward/continue head — a cheap scorer that judges an imagined trajectory before executing it. Adding one is small work with a large safety payoff.

2. Your harness — simulate before acting, the highest-value pattern for agents. The RAP result is the buildable one: use an LLM as both actor and transition function, run a shallow search (MCTS or even beam search, depth 2–3) over predicted outcomes, then execute only the best branch. For any agent that touches something irreversible — sending email, moving money, writing to a production database, filing with a regulator — a two-step lookahead over “what state does this leave the system in?” catches a category of failure that retries and guardrails do not. Concrete thing to build: a “pre-flight simulation” step in your own agent’s action loop, gating anything irreversible.

3. Your automations and business — a counterfactual decision audit. Any logged decision history you own (support tickets, sales sequences, pricing changes, ops routing) is raw material for abduction → action → prediction. Fit a lightweight dynamics model on the logs, then answer “what would have happened if we had routed this differently?” — with the state held fixed, which is exactly what naive A/B comparisons cannot do. This is process mining that answers why, and it’s useful to operations and finance decisions, not just engineering ones. Note the honest limits up front: non-identifiability means you must present a range under alternative plausible dynamics, never a point estimate. That caveat is a credibility asset, not a weakness.

Where to stay skeptical: anyone selling a video generator as a “world model” for a safety-critical application, including your own enthusiasm for foundation world models. The ~45% physical-consistency number and the 5–9-frame degradation are the numbers to quote. Cosmos and Genie are trained on internet video biased toward indoor scenes, driving, and gaming, which underrepresents exactly the contact dynamics that matter for manipulation.

Domains where the survey suggests the paradigm is genuinely mismatched: finance (reflexivity — deploying the model changes the environment, breaking the stationary-POMDP assumption outright) and education (the “environment” is a human learner with non-Markovian, non-stationary internal state). Both are still worth building for, but not with the physical-world architecture copied over unchanged.

Build Your Own (Minimal Recipe)

Smallest thing that captures ~80% of the value, in build order.

Target: a working latent world model on a simple environment, then the counterfactual layer on top. Realistic effort: a weekend for the toy, 2–4 weeks for something demo-ready.

Components, in order:

1. Environment + replay buffer. Use gymnasium with a pixel-observation classic-control task (CarRacing or a DMC task via dm_control). Collect a few thousand random transitions. Do not skip this — you need real data before you can imagine.
2. Encoder. Small conv net → 32-dim latent. Start deterministic; add the stochastic head later. PyTorch, ~30 lines.
3. Dynamics. A GRU cell over (z_t, a_t) → z_{t+1}. This is the RSSM’s deterministic path. ~15 lines. Resist starting with a transformer; you will not have the data to justify it.
4. Reward head. Two-layer MLP z → r. ~5 lines. This is what makes it a world model rather than a video predictor.
5. Train on real data. Teacher-forced: feed the true next observation back each step. Loss = dynamics prediction + reward prediction. Watch one-step error first; only move on when it is low.
6. Imagine. Seed from real states in the buffer, roll forward 10–15 steps without teacher forcing, sum discounted predicted rewards.
7. Actor-critic in imagination. Backprop the imagined return into a small policy net. This is where sample efficiency appears.
8. Counterfactual layer (the part worth selling). Freeze z_t, branch on two different actions, compare the two rollouts. Ten lines of code, and the most interesting output in the whole system.

The two genuinely hard parts:

• Stopping model exploitation. Your policy will find a corner of latent space where predicted reward is enormous and real reward is zero. Mitigations, in order of effort: cap the horizon short (start at 5, MBPO ramps 1→25); train an ensemble of 3–5 dynamics heads and penalize reward by their disagreement (r̃ = r̂ − λ·u(s,a), MOPO-style); truncate rollouts when the state leaves the data support (MOReL-style). Do the horizon cap first, it is one line.
• Keeping the latent from collapsing. Distinct observations mapping to the same code silently destroys everything downstream. Symptom: one-step error looks fine, rollouts are nonsense. DreamerV3’s fixes — KL balancing with free bits, symlog-transformed predictions, percentile return normalization — exist precisely for this and are worth copying verbatim rather than rediscovering.

Reach for: PyTorch; gymnasium / dm_control for environments; the official DreamerV3 repo as a reference implementation (do not start from it, it is production-hardened and hard to read); diffusers only if you get to the pixel-fidelity stage; an off-the-shelf LLM plus a simple MCTS for the RAP-style agent variant, which needs no training at all and is the fastest path to a demo.

Skip entirely at first: the decoder. MuZero proves you do not need it, and it is the single largest source of blurry-reconstruction frustration. Add it only when a human needs to look at the imagined rollout — which, for demos, is admittedly often.

How to Improve It

Five concrete, testable directions. The first three attack the field’s stated bottleneck; the last two attack the survey itself.

1. Attack compounding error with hybrid temporal abstraction, and measure it properly. Hierarchical models (Director, THICK) reduce effective rollout depth by predicting at coarse time steps; state-space models (Mamba-based StateSpaceDiffuser, EDELINE) give O(T) long-context memory. Nobody in the survey has combined them: a hierarchical model whose high level is an SSM and whose low level is a short-horizon diffusion or RSSM. Testable prediction: coherent rollouts an order of magnitude longer at equal per-frame cost. Measure with WorldBench mIoU-versus-frame-index, not FVD.

2. Build the missing unified evaluation, and make it a product. The survey explicitly names this as “one of the most impactful methodological contributions the field could pursue” — a domain-agnostic protocol combining diagnostic physics probes (WorldBench), perceptual decomposition (VBench), and task-performance normalization (IQM + bootstrap CIs from rliable). This is unglamorous, entirely buildable by a small team, and would be cited by everyone. For anyone building on this space it is also a credible reputation play: we measure world models honestly.

3. Test whether counterfactual validity can be certified, not just claimed. Non-identifiability is presented as a hard theoretical wall. But there is an untested engineering response: train an ensemble of dynamics models that all fit the factual data equally well, then report the spread of their counterfactual answers as a validity interval. Wide spread = non-identifiable, do not trust it. Narrow spread = the counterfactual is robust across plausible worlds. Cheap to build, directly usable in decision auditing, and to my reading nobody in the survey has done it.

4. Push the LLM-as-world-model result past its ceiling. RAP works on short symbolic states and degrades on long or continuous ones. The obvious hybrid the survey does not propose: use the LLM as the high-level transition function over a symbolic state (which it handles well) and a small learned latent model as the low-level one (which it handles badly). Two-level world model, two different substrates, joined at the abstraction boundary. Directly relevant to agent orchestration, where the high-level state genuinely is symbolic.

5. Take the reflexivity problem seriously instead of noting it. The survey observes that in finance, deploying a world model changes the world it models, and then moves on. That observation applies to any deployed agent system, including ordinary business automation — your routing model changes the distribution of what gets routed. The survey suggests game-theoretic and multi-agent formulations. Nobody has built one. For anyone already running multi-agent systems, this is the most defensible research direction on the list, because the pain is already familiar.

One thing to attack in the survey’s framing itself: it treats the predictive-model / generative-simulator split as a definitional cleanup. It is more than that. If they are genuinely different objects, they should have different names, different benchmarks, and different claims — and the current confusion is doing active commercial damage, because “world model” on a slide now means “we generate good video” to buyers who hear “we understand physics.”

Glossary

• World model — a learned internal simulator of an environment: given a state and an action, it predicts the next state (and usually reward), so an agent can try things in its head first.
• Latent state (z_t) — a compressed vector standing in for the full observation. A 12,288-dimensional image becomes 32–256 numbers that keep what matters for decisions.
• POMDP — partially observable Markov decision process: the formal setting where the agent never sees the true state, only noisy observations, so it must infer a belief about where it is.
• Model-free RL — reinforcement learning that learns which action pays directly from experience, without ever learning how the environment works. DQN, PPO, SAC.
• Model-based RL (MBRL) — reinforcement learning that learns the environment’s dynamics and uses them to plan or to generate training data.
• RSSM (Recurrent State-Space Model) — PlaNet’s latent design, now standard: split the state into a deterministic recurrent part (carries memory) and a stochastic part (carries uncertainty).
• Imagination / latent rollout — repeatedly feeding the dynamics model its own predictions to produce a trajectory without touching the real environment.
• Compounding error — small per-step prediction errors multiplying over a rollout until the imagined future is nonsense. Grows roughly linearly in rollout length and per-step error.
• Model exploitation — a policy learning to score high on the world model’s mistakes; it looks brilliant in imagination and fails in reality.
• Aleatoric uncertainty — genuine randomness in the environment. More data will not remove it.
• Epistemic uncertainty — the model’s own ignorance about regions it has not seen. More data will remove it. Usually estimated by ensemble disagreement.
• Actor-critic — an RL pattern with two networks: the actor picks actions, the critic estimates how good a state is. Dreamer trains both entirely inside imagination.
• Value function V(z) — expected total future reward from a state. Lets you truncate a rollout early and “bootstrap” the rest instead of simulating forever.
• Discount factor γ — how much future reward is worth relative to now (typically 0.99). Keeps infinite-horizon sums finite.
• Human-normalized score (HNS) — (agent − random) / (human − random). 1.0 means human parity. The standard Atari 100k metric; dominated by outliers, so prefer IQM.
• IQM (interquartile mean) — a robust aggregate that drops the top and bottom 25% of runs. Recommended over mean/median for RL comparisons.
• VAE (variational autoencoder) — an encoder-decoder that learns a distribution over latents rather than a point, giving you built-in stochasticity. The blurry-reconstruction complaint comes from its Gaussian likelihood.
• VQ-VAE / tokenizer — quantizes continuous features into a finite vocabulary of discrete codes, so images can be modeled like text tokens.
• JEPA (Joint-Embedding Predictive Architecture) — LeCun’s design: predict the embedding of the next observation, never the pixels. No decoder, no pixel loss.
• Diffusion model — a generator that learns to reverse a gradual noising process; produces sharp, multi-modal samples but needs several denoising steps per output.
• EDM — a diffusion formulation that stays stable with very few denoising steps (three, in DIAMOND) — the reason diffusion became practical for autoregressive world models.
• MCTS (Monte Carlo Tree Search) — decision-time search that repeatedly rolls out promising branches and backs their values up a tree. MuZero’s planner; also RAP’s.
• Value equivalence — the principle behind MuZero: shape the latent to predict rewards, values, and policies rather than to reconstruct observations.
• Counterfactual / do-operator — reasoning about what would have happened under a different action, with everything else held fixed. do(a') means intervene on the action only.
• Abduction — inferring the hidden causes (latent state plus the specific unobserved randomness) that produced an observed trajectory. Step one of a counterfactual.
• Non-identifiability — several different models fit the observed data equally well but disagree about counterfactuals. Fitting the data better cannot resolve it.
• Sim-to-real gap — the failure of models trained in simulation when deployed physically. Splits into a visual gap (rendering, lighting, sensors) and a dynamics gap (contact, friction, latency).
• Domain randomization — training across randomly varied simulated conditions so the model learns what is invariant rather than what is simulator-specific.
• FVD (Fréchet Video Distance) — the field’s default video-quality metric. Biased toward per-frame appearance over temporal realism; the survey argues it has misdirected research.
• VLA (Vision-Language-Action model) — a single model mapping images plus an instruction directly to robot actions. RT-2, OpenVLA, π0.
• Reflexivity — when deploying a model changes the environment it was modeling. Breaks the assumption that dynamics are fixed; acute in finance and in any deployed agent system.