Why this matters Most joint video–audio models focus on what is heard and when, but not where—yet AR/VR and interactive experiences require stereo sounds whose perceived direction follows visible source motion. This paper centers dynamic spatial correspondence: binding explicit motion tracks to stereo audio so the audio image moves coherently with the video.
Key Findings
- Motion-conditioned stereo synthesis: Proposes StereoBind, which injects motion-track information into audio latents via three mechanisms (Visual Motion Binding, Spatial Track Encoder, Residual Track RoPE) to align stereo spatial cues with visual motion.
- Data and evaluation: Builds StereoWorld-29K (paired motion tracks and rendered stereo targets) and StereoWorldBench to measure audiovisual spatial alignment beyond conventional quality metrics.
- Empirical gains: Reports improved stereo spatial fidelity and audiovisual alignment metrics (ILD-W, SELD-Acc, SMR-Err, AST-Ang, AST-Cal) versus prior joint VA and V2SA baselines while preserving audio–visual quality and synchronization.
Who this is for and trade-offs
Great fit if you develop immersive multimodal content (AR/VR, games, cinematic effects) and need stereo audio that spatially tracks visible sources; the motion-track conditioning makes it suitable when explicit source trajectories are available or can be estimated. Look elsewhere if you only need mono audio, do not have reliable source tracks, or require binaural HRTF-personalized rendering—the approach focuses on spatial coherence tied to motion tracks and uses rendered spatial supervision rather than per-user HRTF calibration.