Pixels record appearances, not the underlying ontology that governs dynamics. Code-as-World flips the representation: instead of treating frames as the primary world model, it induces compact, editable, executable world descriptions (scene programs) that can be simulated, rendered, and verified against real observations. That executable abstraction lets you inspect states, parameters, and mechanisms rather than relying on implicit latent predictors.
Key Findings
- Agentic discovery loop: an LLM-driven agent proposes executable world hypotheses, runs them in a physics engine, renders proxy observations, compares to source video, and iteratively patches programs until they match—turning raw footage into verified, reusable simulatable worlds.
- Executable supervision: verified programs provide exact physical labels (states, dynamics, parameters) that scale to large datasets and are used to fine-tune vision–language models for quantitative physical reasoning.
- Empirical gains and release: fine-tuned Code-as-World-VL checkpoints (4B, 9B) and codebase are released; the approach achieves state-of-the-art performance on QuantiPhy validation in the authors' evaluation.
- Practical pipeline: pairs an interpretable world-evolution module (code) with a video renderer so that knowledge-driven dynamics control long-term consequences while learned video priors supply high-fidelity visuals.
Who it's for and tradeoffs
Great fit if you are a researcher or engineer aiming to (a) extract interpretable physical structure from real videos, (b) generate precise physical supervision for training models, or (c) build editable simulatable environments from footage. Look elsewhere if you need pixel-perfect photorealism without simulation or if you cannot afford a simulation/rendering stack—simulation fidelity and the simulator–real gap limit applicability for high-precision control on real hardware. The method also requires iterated agentic synthesis and verification, which increases engineering complexity compared to purely end-to-end learned predictors.
Method (brief)
The core loop is propose → execute → render → verify → refine. Candidate scene programs encode objects, state variables, and dynamics; a lightweight compiler produces a proxy video which conditions a video model to render observations; verification turns mismatches into counterexamples that guide program patches. Verified worlds become labeled data for supervised training of vision–language models focused on quantitative physical reasoning.