Research map / Video / Video generation
MRSAudio: A Large-Scale Multimodal Recorded Spatial Audio Dataset with Refined Annotations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MRSAudio, a 484-hour recorded multimodal spatial audio dataset (binaural/FOA, video, positions, transcripts, scores, lyrics) across four subsets, and benchmarks five spatial audio tasks.
Problem
Existing multimodal audio datasets are mostly monaural or simulated, lacking real recorded spatial audio aligned with visual, positional, and fine-grained semantic annotations needed for spatial audio understanding and generation.
Approach
A large, real-recorded spatial audio corpus with binaural and ambisonic audio, exocentric and egocentric video, UWB-tracked 3D source positions, and fine-grained annotations (transcripts, phonemes, lyrics, MIDI scores, event labels), plus baseline benchmarks on five spatial tasks.
Key results
- On audio spatialization, BinauralGrad trained on all MRSAudio subsets reaches W-L2 0.305e-3 and MOS-P 3.94, beating the DSP baseline (W-L2 1.691e-3, MOS-P 3.75).
- Mono+SP spatial TTS achieves CER 2.56% and SIM 0.98 on MRSSpeech, close to ground truth CER 2.54%.
- Transformer SELD with FOA and no visual input achieves the lowest ER20 of 0.99 and highest LR_CD of 87.32 on MRSSound.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train BinauralGrad on all four MRSAudio subsets instead of the DSP baseline | Audio spatialization · Audio Spatialization (all subsets) | MOS-P: 3.75 ± 0.11 -> 3.94 ± 0.07 | Helped |
| Generate spatial speech end-to-end instead of using Mono + SP | MRSSpeech · MRSSpeech | ANG Cos: 0.44 -> 0.48 | Helped |
| Use Mono + SP instead of end-to-end ISDrama for singing synthesis | MRSSing · MRSSing | MCD: 3.71 -> 3.19 | Helped |
| Generate spatial music directly from score and spatial cues instead of Make-An-Audio 2 + SP | MRSMusic · MRSMusic | FAD: 4.39 -> 2.45 | Helped |
| Add visual input to ConvNet with FOA input (ablation) | SELD on MRSSound · MRSSound | F20°: 9.33 ± 0.36 -> 13.00 ± 2.72 | Helped |
| Add visual input to Transformer with FOA input (ablation) | SELD on MRSSound · MRSSound | LRCD: 87.32 ± 2.14 -> 46.78 ± 7.62 | Hurt |
| Replace ConvNet with Transformer using FOA and no visual input (ablation) | SELD on MRSSound · MRSSound | ER20°: 1.12 ± 0.02 -> 0.99 ± 0.05 | Helped |
| Replace binaural input with FOA for Transformer without visual input (ablation) | SELD on MRSSound · MRSSound | ER20°: 1.11 ± 0.04 -> 0.99 ± 0.05 | Helped |
| Add visual input to Transformer with binaural input (ablation) | SELD on MRSSound · MRSSound | ER20°: 1.11 ± 0.04 -> 1.01 ± 0.02 | Helped |
Surprising findings
- Visual input gives limited or inconsistent gains for Transformer SELD models, and binaural input underperforms FOA.
- General-purpose Make-An-Audio 2 + spatialization underperforms on FAD and pitch accuracy versus the direct score-conditioned pipeline.
Limitations
- Visual modality is only explored in a limited subset of tasks.
- FOA subset is much smaller than binaural, limiting ambisonic modeling.
- Most benchmarks focus on binaural audio.
Experimental setup
Datasets: MRSAudio MRSLife MRSAudio MRSSpeech MRSAudio MRSSing MRSAudio MRSMusic
Benchmarks: MRSAudio audio spatialization MRSAudio spatial TTS MRSAudio spatial SVS MRSAudio spatial music generation MRSAudio SELD (MRSSound)
Metrics: W-L2 A-L2 P-L2 PESQ STFT MOS-Q MOS-P ANG Cos DIS Cos CER
Baselines: DSP binaural rendering Mono+SP CosyVoice+SP F5-TTS+SP Rmssinger+SP Make-An-Audio 2+SP ConvNet SELD
Builds on: BinauralGrad ISDrama STARSS23 FAIR-Play Make-An-Audio 2
Closest papers by meaning
- Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation (ICLR 2025 · similarity 0.91)
- The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models (NeurIPS 2026 · similarity 0.89)
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.89)
- OmniAudio: Generating Spatial Audio from 360-Degree Video (ICML 2025 · similarity 0.88)
- BAT: Learning to Reason about Spatial Sounds with Large Language Models (ICML 2024 · similarity 0.88)
- Audible World Models: Spatially Aware Sound Generation for 3D Worlds (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.