Research map / 3D vision / NeRF and radiance fields
NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeRAF, which jointly trains a Nerfacto radiance field and a neural acoustic field conditioned on a 3D voxel grid built from the radiance field, to synthesize novel views and binaural/monaural room impulse responses at new positions.
Problem
Synthesizing room impulse responses (RIRs) at novel source-microphone positions from sparse data, while also synthesizing novel views, without needing co-located audio-visual sensors or extra annotations such as acoustic coefficients or meshes.
Approach
Query a NeRF (Nerfacto) to fill a 3D voxel grid with color and density, encode it with a jointly trained 3D ResNet into scene features, and condition a neural acoustic field that predicts log-magnitude STFTs of RIRs over time queries; the radiance and acoustic fields are trained jointly so audio supervision also improves vision in sparse-view scenes.
Key results
- On SoundSpaces, NeRAF reports T60 2.14%, C50 0.38 dB and EDT 0.010 s, versus AV-NeRF's 2.47%, 0.57 dB and 0.016 s, the best prior results in the table.
- On RAF, NeRAF reaches STFT error 0.17 dB versus 0.39 for INRAS and AV-NeRF, and improves C50 and EDT over prior methods.
- Joint audio-visual training improves novel-view LPIPS by 11-15% over Nerfacto on apartments 1 and 2 with 75-150 training images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| grid-conditioned acoustic field (3D grid from NeRF radiance field features) vs no grid (ablation) | NeRAF acoustic field · RAF | T60 (%), C50, EDT: T60 8.15 -> 7.47; C50 0.674 -> 0.609; EDT 0.212 -> 0.0201 | Helped |
| filling grid with zeros instead of NeRF-derived content (ablation) | NeRAF grid · RAF | T60 (%), C50, EDT: T60 7.75 vs 7.47; C50 0.626 vs 0.609; EDT 0.0205 vs 0.0201 | Helped |
| grid resolution 256 instead of 128 (ablation) | NeRAF grid · RAF | T60 (%), C50, EDT: T60 7.42 vs 7.47 (better); C50 0.608 vs 0.609; EDT 0.0204 vs 0.0201 (worse) | Mixed |
| spectral convergence + MSE spectral loss vs plain MSE loss (ablation) | NeRAF loss · SoundSpaces | C50 and EDT (T60 trade-off): C50 0.53 -> 0.38; EDT 0.015 -> 0.010; T60 1.89 -> 2.14 (worse) | Helped |
| SC + MSE spectral loss vs SC + L1 spectral loss (ablation) | NeRAF loss · SoundSpaces | T60 vs C50/EDT: T60 1.56 (L1) vs 2.14 (MSE); C50 0.42 vs 0.38; EDT 0.013 vs 0.010 | Mixed |
| joint audio-visual training (NeRAF) vs vision-only Nerfacto | Nerfacto radiance field · Apartment 1 and 2, sparse views (75-150 images) | PSNR, SSIM, LPIPS: e.g. LPIPS 0.246 -> 0.216 at 75 images (apt 1); PSNR +3.78% at 75 images (apt 1) | Helped |
| joint audio-visual training vs separate training of NeRF and acoustic field (ablation) | NeRAF · Office 4 and Apartment 1 (joint vs separate) | audio T60/C50/EDT and visual PSNR/LPIPS: Office 4 audio identical T60 1.22, EDT 0.007; apartment 1 PSNR 27.70 vs 27.00 | No clear effect |
| multi-scene training of acoustic field with shared ResNet3D vs single-scene training (ablation) | NeRAF · Office 4, frl apartment 4, apartment 2 | T60, C50, EDT: T60 2.29 -> 2.24; C50 and EDT unchanged (0.37, 0.010) | No clear effect |
| NeRAF vs AV-NeRF on RAF (new SOTA) | NeRAF · RAF | T60, C50, EDT, STFT error: STFT error 0.17 vs 0.36 (INRAS) and 0.39 (AV-NeRF); T60 7.47 vs 8.01 | Helped |
| NeRAF vs AV-NeRF on SoundSpaces | NeRAF · SoundSpaces | T60, C50, EDT: T60 2.47 -> 2.14; C50 0.57 -> 0.38; EDT 0.016 -> 0.010 | Helped |
| grid resolution 128 with 4,096-feature setting on RWAVS vs AV-NeRF | NeRAF · RWAVS | overall MAG and ENV: Overall MAG 1.596 (NeRAF) vs 1.504 (AV-NeRF); ENV 0.144 vs 0.145 | Mixed |
| furniture proximity to microphone (ablation) | NeRAF · SoundSpaces apartments and RAF Furnished Room | T60, C50, EDT: Direction varies by scene | No clear effect |
Surprising findings
- Using the MSE spectral loss rather than the L1 spectral loss of prior work gave a better trade-off on C50 and EDT, while the L1 variant gave the best T60.
- Multi-scene training of the acoustic field with one shared model matched single-scene performance (T60 2.24 vs 2.29), though NeRF is still trained per scene.
Limitations
- A NeRF must still be trained separately for each scene, so the approach is not generalizable across scenes.
- Limited to static scenes with static sound sources.
- Performance degrades on RWAVS House scene, attributed to many small objects requiring a higher-resolution grid.
Experimental setup
Datasets: SoundSpaces RAF RWAVS
Benchmarks: SoundSpaces RAF RWAVS
Metrics: T60 C50 EDT STFT error PSNR SSIM LPIPS
Baselines: NAF INRAS AV-NeRF NACF Opus AAC Nerfacto
Builds on: NeRF Nerfacto NAcF AV-NeRF INRAS
Closest papers by meaning
- AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis (NeurIPS 2024 · similarity 0.88)
- SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding (CVPR 2025 · similarity 0.88)
- Resounding Acoustic Fields with Reciprocity (NeurIPS 2025 · similarity 0.88)
- SoundBrush: Sound as a Brush for Visual Scene Editing (AAAI 2025 · similarity 0.87)
- Hearing Anything Anywhere (CVPR 2024 · similarity 0.87)
- Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.