Research map / Label-efficient and robust learning / Recognition and long-tail
RealMAN: A Real-Recorded and Annotated Microphone Array Dataset for Dynamic Speech Enhancement and Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RealMAN, a real-recorded 32-channel microphone array dataset (83.7 h speech, 144.5 h noise) with direct-path clean speech and source location annotations, and benchmarks speech enhancement and localization trained on real vs simulated data.
Problem
Multichannel speech enhancement and source localization models are trained mainly on simulated RIRs and diffuse noise, which mismatch real acoustics; large real-recorded datasets with clean target speech and source location annotations are lacking.
Approach
Record real speech played by a loudspeaker (static and moving) and real ambient noise with a 32-channel array across 32 speech and 31 noise scenes; estimate direct-path clean speech by aligning source speech to recordings with a source-guided network (AlignSpatialNet) and annotate speaker location with a fisheye camera tracking an LED light.
Key results
- Training FaSNet-TAC/SpatialNet on real speech plus real noise gives best results on real test set; SpatialNet real/real reaches MOS-OVR 2.71 and CER 14.5% (static) versus sim/sim 2.46 and 17.9%.
- CRNN localization with real speech and real noise reaches 89.2% ACC(5°) and 2.2° MAE on static speakers versus 60.4% and 7.8° for simulated training.
- A variable-array FaSNet-TAC trained on 2-8 channel sub-arrays achieves MOS-OVR 2.28 on static test, matching the fixed-array model (2.28).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with real speech and real noise instead of simulated speech and simulated noise | SpatialNet · RealMAN test, static and moving speakers | SI-SDR: -5.2 -> 7.3; -4.8 -> 4.3 | Helped |
| Train with real noise instead of simulated noise | CRNN · RealMAN test, static and moving speakers | ACC(5°): 80.0 -> 89.2; 75.8 -> 86.7 | Helped |
| Train with real speech instead of simulated speech, keeping real noise | FaSNet-TAC · RealMAN test, static and moving speakers | MOS-SIG: 2.83 -> 2.79; 2.75 -> 2.73 | No clear effect |
| Train with real noise instead of simulated noise | FaSNet-TAC · RealMAN test, static and moving speakers | SI-SDR: 0.4 -> 2.1; 0.4 -> 1.2 | Helped |
| Train with real noise instead of simulated noise | SpatialNet · RealMAN test, static and moving speakers | SI-SDR: 4.7 -> 7.3; 2.7 -> 4.3 | Helped |
| Train on variable sub-arrays rather than the fixed test array | IPDnet · RealMAN test, moving speaker | ACC(5°): 85.4 -> 78.8 | Hurt |
| Train on variable sub-arrays rather than the fixed test array | IPDnet · RealMAN test, static speaker | ACC(5°): 87.0 -> 87.1 | No clear effect |
| Train on variable sub-arrays rather than the fixed test array | FaSNet-TAC · RealMAN test, static and moving speakers | WB-PESQ: 1.41 -> 1.42; 1.36 -> 1.36 | No clear effect |
Surprising findings
- SpatialNet trained on simulated speech with real noise performs worst among settings (CER 44.2%), indicating a specific sensitivity to unclear simulation mismatches in microphone positions.
- Real ambient noise spatial coherence is highly time-varying and deviates from the diffuse field in many outdoor scenes.
Limitations
- Source speech is played by a loudspeaker rather than real human speakers, so speaker directivity is not captured.
- Direct-path target speech is estimated and not ground truth; the source speech is only about 35 hours of Mandarin, and the source speech itself is close-sourced.
Experimental setup
Datasets: RealMAN WenetSpeech NOISEX-92 gpuRIR-simulated data
Benchmarks: RealMAN
Metrics: WB-PESQ SI-SDR DNSMOS (MOS-SIG, MOS-BAK, MOS-OVR) CER ACC(5°) MAE
Baselines: FaSNet-TAC SpatialNet CRNN IPDnet
Builds on: FaSNet-TAC SpatialNet IPDnet
Closest papers by meaning
- FINALLY: fast and universal speech enhancement with studio-like quality (NeurIPS 2024 · similarity 0.84)
- Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark (CVPR 2024 · similarity 0.83)
- SonicSim: A customizable simulation platform for speech processing in moving sound source scenarios (ICLR 2025 · similarity 0.83)
- ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Prior (ICML 2025 · similarity 0.82)
- TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation (ICLR 2025 · similarity 0.82)
- Rethinking Training Targets, Architectures and Data Quality for Universal Speech Enhancement (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.