Most video-understanding benchmarks mix visual reasoning with linguistic shortcuts and dataset priors, so high scores can hide whether a model truly understands video. This paper argues that evaluation criteria matter more than yet another benchmark: by systematically auditing existing datasets the authors strip away non-visual shortcuts and expose the real, often-missed challenges for Video-LLMs.
Key Findings
- A large portion of benchmark samples (reported ~55%) are solvable without using visual frames or temporal context — meaning many benchmarks overestimate visual understanding. This implies simple text/context priors can dominate reported performance.
- After filtering shortcuts, the remaining video-native examples produce a substantial capability gap: state-of-the-art models perform only marginally above random guessing. This provides a clearer failure signal for future model work.
- The distilled challenge set serves as a controlled testbed to probe which architectural or data-design choices improve robust video understanding, enabling more informative ablations than mixed-purpose benchmarks.
Who it's for and trade-offs
Great fit if you build or evaluate multimodal/video LLMs and need a diagnostic, shortcut-resistant testbed to measure true video reasoning. It helps researchers prioritize model changes that improve temporal and visual perception rather than exploit dataset artifacts. Look elsewhere if you only need broad, high-level leaderboard scores or a very large-scale task benchmark — Video-Oasis focuses on diagnostic clarity over throughput and coverage.
Where it fits
Use Video-Oasis as a filter or sanity-check before reporting improvements on broader video benchmarks. Treat it as complementary: a small-to-moderate curated suite that surfaces where a model's apparent strengths are actually dataset shortcuts.