Peking University's VaLuE Laboratory has open-sourced OmniEcho alongside the OmniEchoBench unified evaluation suite (arXiv: 2609.23407, GitHub: PKU-VaLuE-Lab/OmniEcho). Addressing the longstanding sensory blind spot where embodied agents rely exclusively on vision, OmniEchoBench compiles 197 real-world 3D acoustic environments with First-Order Ambisonics (FOA) audio across 2,972 QA pairs. The authors introduce OmniEcho, a spatially aware omni-modal agent featuring a dedicated FOA spatial encoder that matches conventional vision-language navigation fidelity purely guided by acoustic signals.

Key Takeaways

  • ✓Bridges the sensory void in embodied AI by equipping robots with human-like spatial sound localization capabilities.
  • ✓Releases OmniEchoBench, spanning 197 authentic multi-modal scenes and 2,972 rigorous audio-visual reasoning QA pairs.
  • ✓Integrates a First-Order Ambisonics (FOA) spatial acoustic encoder preserving strict geometric consistency with visual observations.
  • ✓Demonstrates that acoustic-guided embodied navigation can match traditional high-bandwidth vision-language navigation baselines.
  • ✓PyTorch code, procedural audio rendering pipeline, and pretrained checkpoints open-sourced on GitHub.
🔗

Project Links & Resources

Direct Access
Direct access to official project resources and documentation
🐙
OmniEcho GitHub 官方开源仓库
github.com/PKU-VaLuE-Lab/OmniEcho
↗
🌐
ArXiv 论文预印本 (2609.23407)
arxiv.org/abs/2609.23407
↗
📝
Hugging Face Daily Papers 主页
huggingface.co/papers/2609.23407
↗
🐙
GitHub: PKU-VaLuE-Lab/OmniEcho%60%EF%BC%9B
github.com/PKU-VaLuE-Lab/OmniEcho`；
↗
🐙
GitHub: PKU-VaLuE-Lab/OmniEcho%60
github.com/PKU-VaLuE-Lab/OmniEcho`
↗
🔬

In-Depth Technical Analysis

Core Background & Industry Pain Points Real-world robotic deployment (search-and-rescue, eldercare, surveillance) involves crucial events outside the camera's field of view—cries for help behind walls, dripping water pipes, or grinding industrial motor bearings. Yet embodied multimodal models treat audio strictly as 1D mono transcriptions, discarding interaural time and intensity differences (ITD/ILD) and rendering robots acoustically deaf to 3D physical coordinates. ### Architecture Highlights & Internals PKU's VaLuE Lab introduces OmniEcho, coupling spatial acoustics with 3D perception. The pipeline features: (1) a First-Order Ambisonics (FOA) spatial encoder decomposing acoustic waves into 4-channel B-format spherical harmonics (omni pressure plus X/Y/Z directional gradients); (2) a controllable rendering pipeline synthesizing acoustically reverberant environments while preserving geometric consistency with robotic trajectories; and (3) cross-modal attention transformers aligning spatial audio vectors with visual tokens. ### Authoritative Benchmarks & Measured Scores Benchmarked on OmniEchoBench across 197 authentic multi-room environments: OmniEcho cuts 3D direction-of-arrival (DOA) angular localization error by 36.8%, boosting sound-grounded spatial question-answering accuracy by 41.5% over mono baselines. In audio-vision-language navigation across occluded corridors, acoustic-guided goal arrival hit 68.4%, nearly matching traditional visual navigation pipelines. ### Developer Hands-on Guide Robotics engineers can clone https://github.com/PKU-VaLuE-Lab/OmniEcho to access ROS2 microphone array drivers, audio rendering tools, and pre-trained weights on Hugging Face.