Research map / Vision-language and multimodal / Cross-modal learning
Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RAF, a densely captured real-world audio-visual room impulse response dataset with multi-view images and 6DoF tracking, benchmarks neural acoustic field models on it, and proposes a sim2real pretrain-then-finetune approach for few-shot RIR synthesis.
Problem
Existing novel-view acoustic synthesis and RIR generation methods are developed and evaluated on simplified or synthetic data, lacking a real-world dense audio-visual benchmark with precise pose tracking.
Approach
Capture dense real RIRs (36-microphone tower, robotic loudspeaker stand, motion capture) paired with NeRF-reconstructed multi-view images, then adapt 2D neural acoustic field models to 3D, add energy decay loss (NAF++, INRAS++), and use simulator pretraining followed by real-data fine-tuning for few-shot synthesis.
Key results
- INRAS++ performs best on most metrics in the 48 kHz evaluation with STFT error 0.39 dB, C50 error 0.57 dB and T60 error 6.17% (energy decay loss variant), with lightweight architecture and fast inference.
- Sim2Real INRAS++ at 1% training data achieves C50 error 1.86 dB and T60 error 17.31%, versus 2.42 dB and 25.15% for INRAS++ trained from scratch.
- NAF++ and INRAS++ improve over vanilla NAF and INRAS by a large margin on C50, EDT and T60 errors when the energy decay loss is added.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain INRAS++ on simulated data, then fine-tune on sparse real data | INRAS++ · Furnished room, 1% training data | C50 error (dB): 2.42 -> 1.86 | Helped |
| Pretrain INRAS++ on simulated data, then fine-tune on sparse real data | INRAS++ · Furnished room, 1% training data | T60 error (%): 25.15 -> 17.31 | Helped |
| Use 3D rather than 2D bounce-point sampling (ablation) | INRAS++ · Furnished room | C50 error (dB): 0.57 -> 0.53 | Helped |
| Use 3D rather than 2D bounce-point sampling (ablation) | INRAS++ · Furnished room | T60 error (%): 6.21 -> 5.84 | Helped |
| Add visual inputs to the 2D-bounce-point model (ablation) | INRAS++ with 2D bounce points · Furnished room | C50 error (dB): 0.57 -> 0.58 | No clear effect |
| Add speaker orientation embeddings (ablation) | INRAS++ · Furnished room | C50 error (dB): 0.64 -> 0.55 | Helped |
| Increase energy decay loss weight from λ=1.0 to λ=5.0 (ablation) | INRAS++ · Furnished room | C50 error (dB): 0.58 -> 0.49 | Helped |
| Increase energy decay loss weight from λ=1.0 to λ=5.0 (ablation) | INRAS++ · Furnished room | STFT error (dB): 0.39 -> 0.43 | Hurt |
| Add temporal component to NACF (ablation) | NACF · RAF, 48 kHz | T60 error (%): 6.62 -> 7.31 | Hurt |
| Add energy decay loss to INRAS (ablation) | INRAS · RAF, 48 kHz | C50 error (dB): 0.79 -> 0.57 | Helped |
| Add energy decay loss to INRAS (ablation) | INRAS · RAF, 48 kHz | STFT error (dB): 0.36 -> 0.39 | Hurt |
| Add energy decay loss to NAF (ablation) | NAF · RAF, 48 kHz | T60 error (%): 10.08 -> 8.19 | Helped |
Surprising findings
- Opus audio encoding with nearest-neighbor interpolation is on par with several learning-based methods, suggesting the density of captured RIRs makes simple interpolation strong.
- NAF++ achieves good metric results but fails qualitatively in generated RIRs compared with NACF and INRAS++.
Limitations
- Dataset covers only a single physical room in two configurations, limiting generalization across rooms.
- Sim2real relies on a basic shoebox simulator, and a performance gap remains versus full dense training.
- Real-world RIR collection is expensive and time-consuming, making scaling difficult.
Experimental setup
Datasets: VCTK Pyroomacoustics simulated shoebox RIRs
Benchmarks: RAF
Metrics: STFT error C50 error EDT error T60 error parameters storage inference speed
Baselines: NAF INRAS NACF AV-NeRF Linear interpolation Nearest-neighbor interpolation Opus AAC
Builds on: NAF INRAS NACF AV-NeRF Instant NGP
Closest papers by meaning
- Acoustic Volume Rendering for Neural Impulse Response Fields (NeurIPS 2024 · similarity 0.87)
- Deep Neural Room Acoustics Primitive (ICML 2024 · similarity 0.87)
- NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields (ICLR 2025 · similarity 0.87)
- Hearing Anywhere in Any Environment (CVPR 2025 · similarity 0.87)
- SonicSim: A customizable simulation platform for speech processing in moving sound source scenarios (ICLR 2025 · similarity 0.86)
- AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.