Research map / Large language models / Language, safety and interpretability

SonicSim: A customizable simulation platform for speech processing in moving sound source scenarios

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SonicSim, a Habitat-sim-based toolkit for customizable simulation of moving sound sources, and the SonicSet benchmark (90 Matterport3D scenes, LibriSpeech/FSD50K/FMA), showing models trained on it generalize better to real recordings for speech separation and enhancement.

Problem

Speech separation and enhancement under moving sound source conditions lack large, diverse, and acoustically realistic data; real datasets are small and fixed, while synthetic datasets suffer from acoustic mismatch with real environments.

Approach

Use the Habitat-sim acoustic renderer with Matterport3D scenes and material mapping to synthesize moving-source audio with multi-level control (scene, microphone, source), interpolating RIRs along trajectories to create a large synthetic benchmark whose models transfer better to real recordings.

Key results

• Models trained on SonicSet achieved overall the best separation performance on the real-world RealSEP and HumanSEP datasets compared with models trained on LRS2-2Mix and Libri2Mix.
• On SonicSet, Mossformer2 showed the state-of-the-art separation performance among the 11 benchmarked models.
• For speech enhancement on the real-world RealMAN test set, models trained on SonicSet achieved overall the best results across multiple metrics, including CER.

Measured effects reported in the paper

Surprising findings

• Material removal had the largest effect on separation/enhancement performance among tested environmental factors, while room shape had a relatively small effect.

Limitations

• Realism of simulated audio is limited by the level of detail in imported 3D scene models; gaps or incomplete structures degrade reverberation simulation.
• Real-world validation set (RealSEP) is small, about 5 hours recorded with a single microphone and one participant.
• 3D scenes cannot be modified by users; only objects can be added.

Experimental setup

Datasets: SonicSet LibriSpeech FSD50K FMA Matterport3D LRS2-2Mix Libri2Mix RealMAN VoiceBank-DEMAND DNS Challenge WSJ0-2Mix

Benchmarks: SonicSet RealSEP HumanSEP LRS2-2Mix Libri2Mix RealMAN VoiceBank-DEMAND DNS Challenge test set

Metrics: SI-SNR SDR STOI WER CER PESQ NISQA SigMOS DNSMOS

Baselines: Conv-TasNet DPRNN DPTNet SuDORM-RF A-FRCNN SKIM TDANet BSRNN TF-GridNet Mossformer Mossformer2 DCCRN

Builds on: Habitat-sim Libri2Mix WHAMR! Permutation invariant training Mossformer2

Closest papers by meaning

• Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark (CVPR 2024 · similarity 0.86)
• SonicVisionLM: Playing Sound with Vision Language Models (CVPR 2024 · similarity 0.86)
• Massive Sound Embedding Benchmark (MSEB) (NeurIPS 2025 · similarity 0.84)
• Sonic: Shifting Focus to Global Audio Perception in Portrait Animation (CVPR 2025 · similarity 0.84)
• Acoustic Volume Rendering for Neural Impulse Response Fields (NeurIPS 2024 · similarity 0.84)
• V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · similarity 0.84)