A world model trained only on robot trajectories misses the broad visual experience available in video; a video model, meanwhile, can be costly to use for action planning. The central insight here is to use language as a shared interface between these data sources, so one latent-state predictor can learn from both action-conditioned trajectories and visual change.
Key Findings
- A single predictor trained across domains achieves higher average model-predictive-control planning performance than world models trained separately for each domain or task. This suggests that shared training can transfer useful dynamics knowledge rather than requiring a separate model for every setting.
- Adding image and video data improves average planning performance and robustness to different action phrasings, making the system less dependent on one exact wording.
- Against a matched RAEv2-based video-generation baseline, it reaches comparable planning performance with 6× less training cost and plans 13.9× faster.
- The pretrained model can initialize LaLa-Policy, which jointly predicts actions and future visual representations. On LIBERO and RoboCasa, it performs competitively with Cosmos Policy, with inference up to 5.5× faster.
How It Works
Numerical actions and visual changes are expressed in language, allowing heterogeneous datasets to share a training interface. A predictor built on a language model interprets these descriptions while forecasting future latent visual states. Planning then uses those imagined states, rather than requiring full video generation at each step.
Who It Suits—and the Trade-off
A good fit for researchers studying robot learning, model-based planning, or ways to transfer knowledge from video into control. The results are tied to the evaluated planning tasks and benchmarks; they do not establish reliable physical reasoning in arbitrary environments. Teams looking for a ready-to-deploy robot controller, rather than a research model and policy initialization, should look elsewhere.