Most video-generation evaluations focus on single-rollout plausibility (does this video look realistic?) rather than whether repeated samplings recover the true distribution of possible outcomes from the same initial observation and action. This paper makes that gap explicit by formalizing probabilistic alignment — a distributional criterion for world models — and providing an evaluation protocol that turns repeated video rollouts into empirical outcome distributions you can test against analytic references.
Key Findings
- Formalization and tools: PAWBench frames probabilistic alignment as a measurable distributional property and introduces PAWEval, an outcome-level protocol that maps repeated rollouts to empirical distributions over discrete physical outcomes.
- Large-scale evaluation: Across 50 curated scenarios and 11 contemporary video-generation systems, no model consistently matches the reference probabilities while also recovering the full range of valid behaviors.
- Typical failure modes: Models tend to concentrate probability mass on a few outcomes instead of reproducing the reference distribution; many generations are unscoreable when outcomes are ambiguous, reducing effective sample size for distribution tests.
- Intervention tests: The paper studies whether prompts, initial-noise variation, or additional training can reshape predictive distributions, showing limited success and indicating that distributional misalignment is a deep, model-level issue.
Who it's for and trade-offs
Great fit if you care about using video generators as stochastic world models for downstream planning, control, or uncertainty-aware simulation: PAWBench gives a concrete protocol and metrics to test whether repeated samples reflect true environmental uncertainty. Look elsewhere if your priority is single-rollout photorealism or open-ended, unconstrained video creativity; PAWBench emphasizes curated scenarios with known reference distributions and requires many rollouts per condition, so it trades breadth for precise, interpretable calibration tests. The benchmark highlights a clear research gap: improving distributional fidelity, not just per-sample realism.