Most video-language systems focus on describing what happens, not precisely when the supporting evidence occurs. Temporal grounding is inherently a set-valued task (multiple evidence intervals, varying cardinality and fragmentation), so single-pass labels and common RL rewards misalign with the objective. TimeLens2 reframes supervision and optimization around interval sets and matching-free temporal feedback, aiming for generalist grounding across lengths, domains, and query styles.
Key Findings
- Treats temporal evidence as an interval set throughout data curation and training, avoiding brittle one-to-one segment matching.
- Constructs TimeLens2-93K: a large multi-span supervision pool built from caption-derived proposals, independent localization, cross-agent consensus, semantic verification, and boundary refinement to obtain reliable multi-interval labels.
- Introduces a temporal Wasserstein reward that computes the exact 1D W1 distance between uniform distributions over merged interval supports, providing dense, matching-free feedback under unequal cardinalities; pairs this with temporal IoU for precise overlap guidance.
- Delivers consistent gains: the 2B, 4B, and 8B variants significantly improve mIoU over their Qwen3-VL backbones (reported +14.2, +13.0, +18.1 mIoU) and outperform size-matched baselines and much larger open models across seven benchmarks.
Who It's For & Trade-offs
Great fit if you are developing or evaluating multimodal LLMs that must localize multiple evidence spans in long, heterogeneous videos — e.g., generalist video QA, retrieval with temporal grounding, or multimodal agent perception. The approach relies on a curated multi-span supervision pool and heavy training complexity: it is less suitable when only single-clip or single-interval grounding is required, when labeled data is scarce, or when compute/latency constraints prohibit multi-billion-parameter multimodal models. Expect improved generalization to varied query forms at the cost of dataset construction overhead and larger model sizes.
How It Works (method highlights)
- Data: generate caption-derived temporal proposals, then apply independent localizers and multiple agents to reach consensus; verify semantics and refine boundaries to produce robust multi-span labels (TimeLens2-93K).
- Optimization: use a temporal Wasserstein (W1) reward between merged interval supports to provide dense, matching-free supervision even when predicted and ground-truth cardinalities differ; complement with temporal IoU to reward precise overlaps.
- Models: fine-tune multimodal LLM backbones (Qwen3-VL variants) at multiple scales (2B/4B/8B), showing that architecture and reward choices drive substantial mIoU improvements compared with previous pipelines.
Taken together, TimeLens2 is an empirical and methodological push toward treating temporal grounding as a true set prediction problem, with practical recipes for dataset construction and reward design that scale to generalist multimodal LLMs.