Most video generation work either omits audio or synthesizes it as a separate stage, which misses tightly coupled visual–acoustic events. This paper’s core insight is that a compact joint generator and training pipeline can produce high-resolution (2K) clips with native, synchronized audio and video while remaining small enough to release for wider research use.
Key Findings
- Joint 7B generator: A single 7B denoising generator produces modality-specialized audio and video streams. Early layers process streams independently; later layers use Gated Cross-Modal Attention to modulate cross-modal interactions at token and head granularity, improving synchronization without blowing up model size.
- Data & training recipe: A unified Audio-Video Data System builds temporally coherent clips with structured multimodal annotations and capability-oriented pools. Progressive Joint Training (two-stage audio-video pretraining + high-quality finetuning) stabilizes multimodal learning.
- Alignment & refinement: Audio-Video Reinforcement Learning applies modality-aware multimodal feedback to post-train each stream. For high-resolution output, an Autoregressive 1-Step 2K Refinement converts a multi-step teacher into a one-eval-per-chunk student, enabling practical 2K inference.
- Practical release: The paper reports competitive performance against open-source baselines and releases a compact 7B generator plus a 2K Refiner to facilitate reproducibility and follow-up research.
Who it’s for — tradeoffs
Great fit if you are a multimodal researcher or engineer who needs a compact, reproducible baseline for native audio–video generation at high resolution (2K) and wants to experiment with cross-modal attention, RL-based alignment, or 1-step refinement distillation. Look elsewhere if you need very long-horizon video generation, interactive/world-model capabilities, or production-ready pipelines for multi-minute films — the focus here is on clip-level synchronization, modeling recipe, and releasing accessible models rather than full-scale production tooling.
Where it fits
Positioned between research demos that either ignore audio or append it post-hoc and large proprietary systems that produce audio–video jointly but are not publicly available. Its combination of a compact released model, explicit cross-modal gating, and a 2K refinement distillation offers a reproducible middle ground for the community.
Method highlights
- Gated Cross-Modal Attention: head- and token-wise gates let the model control when and how audio and video tokens influence each other.
- Modality-Aware Multimodal Feedback: separate feedback channels for video, audio, and cross-modal signals during RL finetuning to recover alignment lost in earlier distillation steps.
- Autoregressive 1-Step 2K Refinement: distills a multi-step bidirectional refiner into a student that needs one denoising pass per temporal chunk, making 2K inference tractable.
Overall, the paper contributes a compact, well-documented training and inference recipe plus released checkpoints that lower the barrier for experiments on synchronized high-resolution audio–video generation.