Most embodied agents lack a systematic way to evaluate and learn spatial hearing: humans effortlessly localize and combine audio direction with vision, but existing omni-modal models miss a dedicated spatial-audio channel and there is no unified real-world benchmark. This work supplies both a large real-world evaluation suite and a model architecture that explicitly separates semantic and spatial audio pathways, making spatial audio a practical signal for embodied reasoning and navigation.
Key Findings
- OmniEchoBench: a real-world unified benchmark comprising spatial audio-visual perception and sound-guided navigation. It contains 197 spatial scenes with 2,972 QA pairs for perception and 30 real-world environments yielding 900 navigation tasks with dense first-order ambisonics (FOA) recordings.
- Scalable data pipeline: a controllable FOA rendering pipeline that preserves geometric consistency between sound sources, visuals, and agent trajectories for both perception QA and navigation trajectory synthesis.
- Model design: OmniEcho augments a pretrained semantic audio pathway with a dedicated FOA spatial encoder and uses a three-stage alignment training to jointly capture auditory semantics and spatial cues while retaining semantic audio pretraining benefits.
- Empirical results: OmniEcho attains state-of-the-art performance on the spatial audio-visual perception tasks and achieves sound-guided navigation performance approaching some traditional text-guided VLN baselines (example metrics: OmniEcho reported SR 16.2% and SPL 11.5% on OmniEchoBench-Nav), while highlighting persistent challenges in fine-grained localization, distance estimation, path efficiency, and reliable stopping.
Who it's for and tradeoffs
Great fit if you research multimodal embodied perception or want a real-world benchmark and baseline model to study spatial audio integration in navigation and QA. The dataset and model are valuable when you need FOA spatial audio, geometric consistency between audio and visuals, and reproducible navigation tasks. Look elsewhere if your focus is purely simulated binaural audio, large-scale web-scale audio-only tasks, or if you need out-of-the-box high-efficiency navigation systems — the work prioritizes spatial-audio understanding and diagnostic evaluation over immediate state-of-the-art VLN path efficiency.