Research map / 3D vision / 3D generation
SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Synthesizes binaural ambient sound at novel viewpoints from sparse pre-recorded microphone references, using visual-acoustic binding embeddings (RT60 from panoramic RGB-D) for reference placement and transformer-based reference weighting.
Problem
Novel-view acoustic synthesis of ambient sound in arbitrary real scenes from sparse reference recordings, without knowing individual sound sources or room impulse responses, and with limited reference microphone budgets.
Approach
A visual-acoustic binding encoder pretrained to predict RT60 from panoramic RGB-D, used both to cluster candidate locations for reference microphone placement and as keys/queries in a transformer that weights each reference's contribution conditioned on the target viewpoint, feeding a U-Net binaural renderer.
Key results
- On Soundspace-Ambient unseen scenes, SoundVista (4 refs) achieves STFT 2.517 vs. 3.438 for ViGAS (best single-reference baseline), a 32.5% improvement on the seen-scene setting.
- On the N2S real-world benchmark, SoundVista reaches STFT 1.073 vs. 1.201 for ViGAS, and LRE 0.776 vs. 0.840 for AV-NeRF.
- VAB-based reference sampling improves nearest-GT STFT in unseen scenes from 4.680 (w/o VAB) to 3.835 (vis+location).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SoundVista instead of Interp GT | SoundVista · Soundspace-Ambient seen scenes | STFT: 3.179 -> 2.442 | Helped |
| use SoundVista instead of Interp GT | SoundVista · Soundspace-Ambient unseen scenes | STFT: 3.415 -> 2.514 | Helped |
| use SoundVista instead of ViGAS | SoundVista · N2S | STFT: 1.201 -> 1.073 | Helped |
| add visual input (ablation) | SoundVista · Soundspace-Ambient seen scenes | STFT: 3.228 -> 2.442 | Helped |
| add adaptive reference reweighting (ablation) | SoundVista · Soundspace-Ambient seen scenes | STFT: 2.908 -> 2.442 | Helped |
| add renderer pretraining (ablation) | SoundVista · Soundspace-Ambient seen scenes | LRE: 1.877 -> 1.390 | Helped |
| use decoupled global and local conditions (ablation) | SoundVista · Soundspace-Ambient seen scenes | STFT: 3.017 -> 2.442 | Helped |
| include waveform loss (ablation) | SoundVista · Soundspace-Ambient seen scenes | STFT: 3.422 -> 2.442 | Helped |
| include scaled spectrogram magnitude loss (ablation) | SoundVista · Soundspace-Ambient seen scenes | MAG: 0.301 -> 0.289 | Helped |
| include ILD loss (ablation) | SoundVista · Soundspace-Ambient seen scenes | LRE: 1.420 -> 1.390 | Helped |
| use depth instead of location only (ablation) | VAB · Matterport3D novel scenes, w/o finetune | RT60 error: 0.170 -> 0.082 | Helped |
| use rgb+depth instead of location only (ablation) | VAB · Matterport3D novel scenes, w/ finetune | RT60 error: 0.067 -> 0.036 | Helped |
Surprising findings
- More references do not always help: Few-shotRIR did not improve with more references and was sometimes worse than single-reference baselines, attributed to audio-content-dependent integration.
- Depth alone gives most of the RT60 prediction gain over location-only, and rgb+depth is best only after finetuning.
Limitations
- Binaural ground truth for arbitrary target orientations is unavailable, so comparisons rely on nearest/interpolated GT proxies.
- Deep-learning baselines underperform non-learning methods on the complex multi-source benchmark, and the method relies on content-independent weighting that may not capture all cases.
- Evaluation on a single real room for the N2S benchmark, limiting generality claims.
Experimental setup
Datasets: Soundspace-Ambient (Matterport3D + SoundSpaces) N2S real-room benchmark Matterport3D SoundSpaces
Benchmarks: Soundspace-Ambient N2S
Metrics: STFT distance MAG ENV LRE
Baselines: Nearest GT Interp GT AV-NeRF Few-shotRIR VAM BEE ViGAS
Builds on: ViGAS BEE Few-shotRIR AV-NeRF VAM
Closest papers by meaning
- NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields (ICLR 2025 · similarity 0.88)
- Audible World Models: Spatially Aware Sound Generation for 3D Worlds (NeurIPS 2026 · similarity 0.88)
- SoundBrush: Sound as a Brush for Visual Scene Editing (AAAI 2025 · similarity 0.88)
- AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis (NeurIPS 2024 · similarity 0.87)
- Hearing Anything Anywhere (CVPR 2024 · similarity 0.87)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.