One-sentence Summary

Researchers from Peking University, Alibaba Group, and Tsinghua University introduce OmniEchoBench, a six-task benchmark spanning 197 real-world spatial audio-visual scenes, 2,972 question-answer pairs, and 900 first-order ambisonics navigation samples, alongside OmniEcho, a spatially aware omnimodal model that adds an FOA spatial encoder to a pretrained semantic audio pathway, achieving state-of-the-art spatial audio-visual perception and sound-guided navigation performance close to that of traditional vision-language navigation.

Key Contributions

• Introduces OmniEchoBench, a unified benchmark for spatial audio-visual perception and sound-guided navigation, with six tasks over 197 real-world scenes, 2,972 question-answer pairs, and 900 navigation samples using first-order ambisonics audio from 30 environments.
• Provides a controllable spatial audio rendering pipeline that preserves geometric consistency among sound sources, visual observations, and agent trajectories to enable scalable supervision.
• Proposes OmniEcho, an omni-modal model combining an FOA spatial encoder with a pretrained semantic audio pathway; experiments show state-of-the-art spatial audio-visual perception and sound-guided navigation performance close to traditional vision-language navigation, while fine-grained localization and distance estimation remain open challenges.

Introduction

Embodied agents need rich multimodal signals, but spatial audio remains largely underexplored even though it helps humans localize occluded or out-of-view targets. Existing vision-language navigation models are primarily vision-text driven, audio-visual navigation benchmarks often depend on rendered acoustics rather than real-world recordings, and collecting real spatial-audio data is costly because the signal depends on source and listener poses, scene geometry, reflections, and reverberation. The authors introduce OmniEchoBench, a real-world benchmark unifying spatial audio-visual perception and sound-guided navigation, along with a controllable spatial-audio rendering pipeline for scalable training data. They also propose OmniEcho, a spatially aware omni-modal model built on Qwen3-Omni that adds an FOA encoder and a three-stage training procedure to integrate semantic and spatial audio understanding with visual and language inputs.

Dataset

Dataset description

The authors introduce two real-world benchmark datasets, OmniEchoBench-QA and OmniEchoBench-Nav, plus a synthetic training data pipeline built on a unified first-order Ambisonics, or FOA, spatial audio rendering framework.

1. OmniEchoBench-QA

• Source and composition: Collected from real human performances following authored scripts. Each recording captures a first-person main-view video, a 360-degree panoramic video, and 4-channel FOA audio in ACN/SN3D format.
• Scale: 2,972 QA pairs in total.
• Subsets:
  • Fine-grained temporal spatial reasoning: 2,372 QA pairs covering source direction (1,069), 3D localization (521), source motion (473), and camera rotation (309).
  • Bird’s-eye audio-driven localization: 600 QA pairs using an ego-centered top-down map and FOA clips from navigation scenes. The model selects one of four candidate locations labeled A, B, C, and D.
• Filtering and annotation: Scripts deliberately emphasize off-screen and boundary-crossing sound sources so that visual input alone is insufficient. The panoramic video is used only during annotation; at test time the model receives the single-view video or still image plus FOA audio. Annotators label every active sound source at 2 Hz, producing dense spatio-temporal trajectories, and rule-based sampling then creates typed questions and ground-truth answers.

2. OmniEchoBench-Nav

• Source and composition: Real-world indoor navigation data from 30 real-scanned indoor environments. This includes 10 single-room scenes and 20 multi-room or cross-room layouts spanning two to four connected rooms. Each environment is provided as a textured mesh, in .glb format, with a top-down floor render.
• Scale: 30 navigation samples per environment, for 900 samples total. The benchmark includes 12,900 FOA recordings, each 4-channel and 48 kHz, captured over a dense receiver grid.
• Source categories: 505 samples use non-speech environmental events such as mechanical sounds, alarms, doorbells, animal sounds, and footsteps; 395 samples use spoken commands such as help requests and calls.
• Metadata and schema: Each source is annotated with source device, room, semantic description, and a natural-language navigation target description.
• Usage: At test time the agent receives spatial audio from the receiver position nearest to its current location, with directional remapping according to orientation. The agent must infer the sound source location and navigate toward it.

3. Synthetic training data

• The authors use a unified FOA rendering framework. Each sounding source is represented by a time-varying 3D trajectory, and the listener is co-located with the camera. Source stems are encoded into 4-channel FOA audio using real spherical harmonics, with source directions transformed into the listener frame and optional distance attenuation or room effects.
• Spatial audiovisual QA data: Dynamic scenes are generated with Seedance and combined with trajectory-grounded sound events. FOA audio and QA pairs are then synthesized from these scenes.
• Audio-only data: Additional audio-only training data are synthesized to support FOA encoder pretraining and QA.
• Navigation data: Existing VLN trajectories are augmented with destination-conditioned sound events, and FOA observations are rendered along sampled trajectories.
• Resulting supervision: The datasets provide synchronized visual observations, spatial audio, and geometry-grounded supervision for both perception and navigation tasks.
• Cropping and split details: No separate cropping pipeline is described in this section. The main view restriction is that the panoramic video is used only for annotation, while the main-view video or still image is used at test time. Explicit training split ratios or mixture ratios are not stated in the provided section.

Method

The authors construct their benchmark and training data using a unified First-Order Ambisonics spatial-audio rendering framework. In this setup, each sounding source is defined by a time-varying 3D trajectory, while the listener is colocated with the camera in a right-handed frame. Given the source trajectories and listener pose, each monophonic source stem is encoded into FOA using real spherical harmonics:

where $s_{i}(t)$si​(t), $r_{i}(t)$ri​(t), and $g_{i}$gi​ denote the source stem, source-listener distance, and source gain, respectively. Source directions are transformed into the current listener frame to maintain spatial alignment with visual observations.

This unified formulation supports two primary training-data pipelines. For spatial audiovisual question answering, the authors generate dynamic scenes and combine visual content with trajectory-grounded sound events to synthesize FOA audio and QA pairs. They also synthesize audio-only training data to support FOA encoder pretraining. For sound-guided navigation, existing Vision-and-Language Navigation trajectories are augmented with destination-conditioned sound events, and corresponding FOA observations are rendered along sampled trajectories.

To unify spatial audio-visual perception and navigation within a single embodied-agent framework, the model is designed to handle distinct task formulations. For perception tasks, the model takes FOA audio $A$A, a video sequence $V$V or a single image $I$I, and a question $q$q to predict an answer $a$a. For navigation tasks, at each decision step, the model receives a language instruction $L$L, $T$T frames captured by a forward-facing camera, and the FOA spatial audio $A_{t}$At​ within the current temporal window to predict a trajectory over the next $K=8$K=8 time steps.

The core of the method lies in the spatial audio encoding process, which is executed in three stages. In the first stage, the authors pretrain a lightweight FOA encoder $f_{\theta}$fθ​ that maps an FOA clip into a temporal sequence of 384-dimensional tokens. Each clip is converted into a 5-channel input map comprising the log-mel spectrogram of the omnidirectional channel, three active-intensity components, and diffuseness. The encoder is trained to align its representations with a frozen CLIP text encoder using a SigLIP objective, establishing an open-vocabulary sound-semantic space. In the second stage, this semantically aligned encoder is fine-tuned for query-conditioned sound-source localization.

In the final stage, the frozen FOA encoder $f_{\theta^{\star}}$fθ⋆​ is integrated into the Qwen3-Omni backbone.

Each FOA clip is processed along two complementary paths. First, its omnidirectional channel is down-mixed to mono and fed to the original, frozen audio tower to preserve the native semantic audio tokens. Second, the same clip is passed through the FOA encoder to obtain spatial tokens. These spatial tokens are temporally resampled to the 7 Hz audio-token grid and mapped into the language embedding space by a trainable projector. The projected spatial tokens occupy reserved placeholder embedding rows and are inserted immediately after the corresponding audio tokens in temporal order. During this integration, only the large language model parameters and the projector are updated, while the FOA encoder, audio tower, and visual tower remain frozen.

Experiment

The experiments evaluate an omni-modal spatial audio-visual model on two newly introduced benchmarks, OmniEchoBench-QA for spatial-audio question answering and OmniEchoBench-Nav for sound-guided navigation, after training on synthesized FOA-rendered audiovisual and navigation data. Results show that the proposed FOA encoder improves spatial reasoning over the native audio backbone, with source direction, localization, and motion tasks solved mainly from spatial audio while camera rotation depends more on visual cues. Navigation experiments indicate that sound-guided navigation can approach earlier text-guided baselines in real-world scenes, though reliable stopping and efficient navigation remain challenging, and ablations confirm that dedicated FOA pretraining and feature representations are critical for robust performance.

OmniEcho with four-channel FOA audio attains the highest overall accuracy among the reported audio-only models and consistently surpasses the single-channel Qwen3-Omni backbone on spatial reasoning tasks. The gains are strongest for source-centric skills such as direction, localization, and motion, whereas camera rotation remains difficult and more dependent on visual evidence. Real-world spatial audio understanding remains a limitation for models such as SO-7B that lack complementary visual supervision. OmniEcho outperforms all compared audio-only models in overall accuracy and across the reported spatial tasks. Using FOA audio instead of single-channel audio improves performance over the Qwen3-Omni backbone on source direction, localization, and motion. Camera rotation is the weakest area relative to random chance and degrades substantially without visual input.

On the OmniEchoBench-Nav benchmark, the model using FOA audio guidance achieves success and path efficiency close to earlier text-guided VLN baselines, outperforming Seq2Seq and CMA but trailing InternVLA-N1. The binaural audio baseline performs markedly worse on all metrics, suggesting that simulated audio models struggle in this real spatial-audio setting. Despite competitive success, audio-guided navigation still shows weaker stopping and efficiency than the strongest text-guided model. The FOA audio model surpasses the conventional text-guided Seq2Seq and CMA baselines in success rate and SPL, while remaining below InternVLA-N1. The binaural audio baseline records the lowest success rate and SPL among reported models, and the FOA audio model has the longest trajectory length, indicating efficient navigation and reliable stopping remain challenging.

On simulated VLN-CE R2R validation, OmniEcho with audio instructions slightly exceeds earlier text-guided baselines in success rate and improves navigation error, though its path efficiency remains below CMA. It still trails stronger text-guided systems by clear margins in both success rate and path efficiency. Overall, spatial-audio guidance is competitive with early text-guided navigation models but leaves a notable gap to state-of-the-art text-guided approaches. OmniEcho achieves a slightly higher success rate than Seq2Seq and CMA under agent stopping, suggesting audio guidance can match early text-guided VLN baselines in task completion. Its navigation error is lower than Seq2Seq and CMA, and its trajectory length is the shortest among models reporting path length. OmniEcho remains substantially behind InternVLA-N1, NAViLA-SAGE, and VLN-R1 in success rate and SPL, indicating a clear efficiency gap relative to stronger text-guided systems.

The strongest configuration combines the native audio encoder with a frozen FOA encoder and keeps the FOA encoder frozen during Stage 3, achieving the best overall score. Adding the frozen FOA encoder improves most spatial tasks over the native-audio-only baseline, while dropping the native audio encoder or allowing the FOA encoder to update during Stage 3 lowers overall performance. These results show that native semantic audio and dedicated spatial encoding are complementary, and that freezing helps preserve spatial representations. Combining native audio with a frozen FOA encoder yields the highest overall performance among the compared configurations. Adding the frozen FOA encoder to the native-audio-only setup improves overall and most subtasks, with only Camera Rotation decreasing slightly. Training the FOA encoder during Stage 3 instead of freezing it degrades every subtask and overall performance.

Across benchmarks, OmniEcho with four-channel FOA audio consistently improves spatial reasoning over single-channel and other audio-only models, especially for direction, localization, and motion, while camera rotation remains difficult without visual input. In navigation, FOA audio guidance is competitive with early text-guided VLN baselines on OmniEchoBench-Nav and simulated VLN-CE R2R, though binaural audio performs much worse and the model still trails stronger text-guided systems in success and path efficiency. Ablations show that combining the native audio encoder with a frozen FOA encoder yields the best results, indicating that semantic audio and dedicated spatial encoding are complementary and that freezing helps preserve spatial representations.