Research map / Vision-language and multimodal / Cross-modal learning
AV-RIR: Audio-Visual Room Impulse Response Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AV-RIR, a neural codec-based audio-visual model that estimates room impulse responses from reverberant speech and panoramic images, using Geo-Mat material/depth features, speech dereverberation as an auxiliary task, and CRIP image-to-RIR retrieval for late reverberation.
Problem
Estimating the room impulse response (RIR) from reverberant speech is hard for audio-only methods, which struggle with late reverberation, and visual-only methods lack speech information; the paper targets accurate RIR estimation for speech processing and AR/VR.
Approach
A SoundStream-style residual vector quantized codec with dual branches: an RIR branch fused with Geo-Mat features (absorption coefficients from a language-guided material pipeline plus monocular depth) and a speech dereverberation branch fused with panoramic image features, trained jointly; plus CRIP, a CLIP-style panoramic-image-to-RIR retrieval module that replaces late RIR components at inferen
Key results
- AV-RIR outperforms the audio-only S2IR-GAN by 36% on T60, 42% on DRR, 63% on EDT, 89% on early MSE and 98% on late MSE.
- In the SLP task on SoundSpaces, AV-RIR reaches 4.17% WER and 2.02% EER versus 3.54% WER and 3.11% EER for AdVerb, which is the best baseline on each metric.
- In perceptual evaluation, 56% to 79% of participants judged AV-RIR output closer to ground truth than Image2Reverb and VAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add audio-visual inputs and Geo-Mat features to S2IR-GAN-style RIR estimation | AV-RIR · SoundSpaces | T60 Error (ms): 63.1 -> 40.2 | Helped |
| Add audio-visual inputs and Geo-Mat features to S2IR-GAN-style RIR estimation | AV-RIR · SoundSpaces | EDT Error (ms): 168.3 -> 62.1 | Helped |
| Add audio-visual inputs to audio-only AV-RIR (ablation) | AV-RIR · SoundSpaces | WER (%): 5.24 (+36%) -> 4.17 (+49%) | Helped |
| Add CRIP retrieval for late reverberation (ablation) | AV-RIR · SoundSpaces | LMSE (x10−5): 42 -> 6 | Helped |
| Add Geo-Mat features (ablation) | AV-RIR · SoundSpaces | EDT Error (ms): 74.1 -> 62.1 | Helped |
| Add multi-task learning with speech dereverberation (ablation) | AV-RIR · SoundSpaces | T60 Error (ms): 77.8 -> 40.2 | Helped |
| Add multi-task learning with speech dereverberation (ablation) | AV-RIR · SoundSpaces | WER (%): 4.57 (+44%) -> 4.17 (+49%) | Helped |
| Add STFT loss (ablation) | AV-RIR · SoundSpaces | T60 Error (ms): 59.4 -> 40.2 | Helped |
| Add MEL loss (ablation) | AV-RIR · SoundSpaces | WER (%): 4.44 (+46%) -> 4.17 (+49%) | Helped |
| Replace the environment image with a random image (ablation) | AV-RIR · SoundSpaces | WER (%): 4.85 (+41%) -> 4.17 (+49%) | Hurt |
Surprising findings
- CRIP-only retrieval from a panoramic image performs poorly on early components, yet adding its late components to the audio-visual estimate substantially reduces late-reverberation error.
- The AV-RIR speech-dereverberation branch does not beat AdVerb on ASR WER, despite leading on speaker verification EER.
Limitations
- Assumes stationary single-talker input speech or single-source audio without noise.
- Only single-channel input; multi-channel RIR estimation is future work.
- Does not handle noisy, multi-source environments or moving sources.
Experimental setup
Datasets: SoundSpaces LibriSpeech Matterport3D AVSpeech MP3D-based synthetic RIR datastore
Benchmarks: SoundSpaces AVSpeech
Metrics: T60 error DRR error EDT error MSE early/late WER EER RTE
Baselines: Image2Reverb FAST-RIR++ FiNS S2IR-GAN VAM VIDA AdVerb MetricGAN+
Builds on: SoundStream HiFi-GAN CLIP S2IR-GAN FAST-RIR
Closest papers by meaning
- Hearing Anywhere in Any Environment (CVPR 2025 · similarity 0.88)
- Deep Neural Room Acoustics Primitive (ICML 2024 · similarity 0.86)
- Acoustic Volume Rendering for Neural Impulse Response Fields (NeurIPS 2024 · similarity 0.85)
- Hearing Anything Anywhere (CVPR 2024 · similarity 0.85)
- NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields (ICLR 2025 · similarity 0.83)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.