Research map / Video / Video generation
OmniAudio: Generating Spatial Audio from 360-Degree Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces 360V2SA, the task of generating first-order Ambisonics spatial audio from 360-degree video, with the Sphere360 dataset and the OmniAudio flow-matching model using coarse-to-fine self-supervised pre-training and dual-branch panoramic/perspective video conditioning.
Problem
Existing video-to-audio generation produces non-spatial audio and uses perspective video with limited field of view, so it misses the spatial cues needed for 3D sound. Paired 360-degree video and spatial (FOA) audio data is also scarce.
Approach
Generate FOA audio directly from 360-degree video with a latent flow-matching DiT, pre-trained in a coarse-to-fine self-supervised way (masked token prediction on converted non-spatial audio, then FOA-only), and conditioned on a dual-branch video representation (equirectangular global features plus perspective local features from a frozen MetaCLIP-Huge encoder).
Key results
- On Sphere360-Bench, OmniAudio reaches FD 88.30 and KL 1.58, versus 271.15 / 2.39 for MMAudio + AS.
- OmniAudio's angular DoA error is 1.28 on Sphere360, compared with 1.49 for ViSAGe (FoV) and 1.51 for ViSAGe (360).
- Human MOS-SQ and MOS-AF on Sphere360-Bench are 84.67 and 87.23 for OmniAudio, above all baselines (best baseline MMAudio + AS at 75.34 and 77.56).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add spatial-audio pre-training to non-spatial-audio pre-training (ablation) | OmniAudio · Sphere360 | FD: 97.26 -> 88.30 | Helped |
| add non-spatial-audio pre-training to spatial-audio pre-training (ablation) | OmniAudio · Sphere360 | FD: 97.57 -> 88.30 | Helped |
| add self-supervised pre-training (ablation) | OmniAudio · Sphere360 | FD: 104.57 -> 88.30 | Helped |
| use ERP rather than EAC in the panoramic branch alongside perspective video (ablation) | OmniAudio dual-branch video input · Sphere360 | FD: 89.89 -> 88.30 | Helped |
| add panoramic ERP video to perspective-video input (ablation) | OmniAudio · Sphere360 | FD: 88.80 -> 88.30 | Helped |
| add perspective video to EAC video input (ablation) | OmniAudio · Sphere360 | FD: 93.37 -> 89.89 | Helped |
| add perspective video to ERP video input (ablation) | OmniAudio · Sphere360 | FD: 97.83 -> 88.30 | Helped |
| increase model size from 472M to 1.2B parameters (ablation) | OmniAudio · Sphere360 | FD: 104.19 -> 88.30 | Helped |
| increase model size from 291M to 1.2B parameters (ablation) | OmniAudio · Sphere360 | FD: 108.50 -> 88.30 | Helped |
Surprising findings
- Results on the out-of-distribution YT360 test set show MMAudio + AS with a lower KL (1.71 in the table) than the text reports (1.51), an inconsistency in the paper.
- Ablation on the Large model reports Delta_Angular 1.26 in the model-size table, but the main table reports 1.28 for the same model.
Limitations
- Struggles with scenes containing many sound-emitting objects, misclassifying event types (e.g., instrument sounds as applause).
- Current cleaned dataset (about 100K samples) is judged insufficient for robust real-world 360V2SA.
- Subjective MOS scores show large standard deviations, reflecting the subjectivity of spatial audio evaluation.
Experimental setup
Datasets: Sphere360 YT360 FreeSound AudioSet VGGSound
Benchmarks: Sphere360-Bench YT360-Test
Metrics: FD KL divergence DoA theta error DoA phi error DoA spatial angle error MOS-SQ MOS-AF
Baselines: MMAudio + AS Diff-Foley + AS ViSAGe (FoV) ViSAGe (360)
Builds on: Stable Audio Voicebox Audiobox ViSAGe MMAudio
Closest papers by meaning
- ViSAGe: Video-to-Spatial Audio Generation (ICLR 2025 · similarity 0.89)
- Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation (ICLR 2025 · similarity 0.89)
- MRSAudio: A Large-Scale Multimodal Recorded Spatial Audio Dataset with Refined Annotations (NeurIPS 2025 · similarity 0.88)
- Audible World Models: Spatially Aware Sound Generation for 3D Worlds (NeurIPS 2026 · similarity 0.87)
- AV-Cloud: Spatial Audio Rendering Through Audio-Visual Cloud Splatting (NeurIPS 2024 · similarity 0.85)
- Tri-Ergon: Fine-Grained Video-to-Audio Generation with Multi-Modal Conditions and LUFS Control (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.