Most text-to-video systems infer temporal dynamics implicitly from compressed prompts, which often yields physically inconsistent motion. VideoCoCo takes a different tack: it makes the temporal process explicit and executable by generating Blender code as a process-level chain-of-thought, rendering a deterministic draft, then conditioning a generative video editor on that draft to produce the final photorealistic video. This separation lets the system control and inspect dynamics independently from visual realization.
Key Findings
- Executable code as CoT: Synthesizing Blender programs gives a fully-specified, inspectable spatiotemporal draft (scene + scripted evolution) that a sandboxed simulator deterministically renders, enabling precise process-level control rather than relying solely on learned priors.
- Dual-engine decomposition: A coding agent (program synthesis) + simulation engine (deterministic draft) + instruction agent + adapted generative editor (draft-conditioned editing) separates reasoning about dynamics from high-fidelity rendering, reducing ambiguity in temporal evolution.
- Empirical gains: Introducing VideoCoCo to an OmniWeaving base raised PhyGenBench consistency from 0.475 to 0.558 and VBench-2.0 average from 52.18% to 77.88%, with notable improvements on mechanics, material, and thermal categories.
- Supervised adaptation: VideoCoCo-3K (draft–instruction–target triplets) adapts the editor to realize simulated drafts as photorealistic outputs while preserving the encoded spatiotemporal process.
Who it's for and tradeoffs
Great fit if you need reproducible, controllable, and physically plausible text-to-video synthesis—researchers benchmarking physical plausibility, systems that must enforce specific dynamics, and workflows that value inspectable intermediate artifacts. It is less appropriate when you only need stylistic or purely artistic video generation without fidelity to real-world physics, or when computational and engineering budgets cannot accommodate simulation and program synthesis overhead. The approach inherits simulator limits (scenes must be expressible in Blender and fidelity depends on simulation detail) and adds engineering complexity compared with end-to-end generative models.
Where it fits
VideoCoCo occupies the intersection of programmatic simulation and generative video editing: it complements appearance-driven generative models by providing an explicit dynamics reference, improving categories where learned priors struggle (materials, thermal effects, mechanics) while relying on a separate editor for photoreal rendering.