Most dense video captioning systems rely on strict token-by-token autoregressive decoding, which becomes a bottleneck as videos grow longer and contain more events. This paper's key insight is that tokens across distinct temporal events often exhibit weak dependencies, so the causal graph can be restructured: preserve sequential decoding within events but decode across events in parallel, guided by compact event-level plans.
Key Findings
- Parallelized autoregressive pipeline: a two-stage approach first autoregressively generates compact global event tokens to capture inter-event causality, then decodes event-conditioned token subchains in parallel while retaining intra-event sequential coherence. This keeps global causality without full token-level serialization.
- Empirical gains: matches or surpasses recent SOTA in temporally grounded captioning metrics while delivering substantial inference speedups (reported ≈3.7–3.8× wall-time decoding speedup versus a competitive omni-modal LLM baseline) and strong generalization across related temporal grounding tasks.
- Lightweight implementation choices: built on a 3B omni-LLM backbone with targeted training (LoRA on subset data) and modular audio–visual alignment, demonstrating that model-scale reduction can be offset by architectural decoding changes.
Who it's for and trade-offs
Great fit if you need denser, temporally grounded natural-language summaries of long videos where decoding latency matters (e.g., large-batch offline captioning, event retrieval pipelines). Look elsewhere if your application demands strictly online token streaming with minimal buffering across events or if extreme simplicity matters—restructuring the dependency graph and training the latent planner adds design complexity and some supervision needs.
Method snapshot
The method hinges on (1) a latent global planning mechanism that autoregressively emits a small sequence of event tokens G^1...G^K to encode event-level ordering and aggregated AV semantics, and (2) an event-factorized parallel decoder that blocks cross-event token attention but grants all subchains access to the shared global tokens, enabling independent parallel generation of per-event captions while preserving inter-event context.