Research map / 3D vision / Gaussian splatting
AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AV-GS, which uses an explicit 3D Gaussian Splatting scene representation with learnable audio-guidance parameters and audio-aware point densification/pruning to condition a binauralizer for novel view acoustic synthesis.
Problem
Novel view acoustic synthesis (rendering binaural audio at arbitrary listener poses from mono source audio) is limited by slow NeRF-based conditioning and by weak modeling of room geometry, materials, and listener-source spatial relations.
Approach
Decouple visual geometry from audio by building an audio-focused point representation on 3DGS points, with a learnable audio-guidance parameter per Gaussian (initialized from SH and rotation), decoded through an acoustic field MLP using listener/source-relative directions, and grown/pruned by per-point audio gradient contribution.
Key results
- On RWAVS, AV-GS reaches overall MAG 1.417 and ENV 0.140, versus AV-NeRF's 1.504 and 0.145 (about 5.7% and 3.4% relative improvement).
- On SoundSpaces RIR generation, AV-GS reports T60 2.23, C50 0.53, EDT 0.014, the best among compared methods including AV-NeRF (2.47, 0.57, 0.016).
- Inference time for rendering the condition is 0.08s versus 1.4s for AV-NeRF with one view and 7.6s with four views.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace AV-NeRF with AV-GS | binaural audio synthesis · RWAVS overall | MAG: 1.504 -> 1.417 | Helped |
| replace AV-NeRF with AV-GS | binaural audio synthesis · RWAVS overall | ENV: 0.145 -> 0.140 | Helped |
| replace AV-NeRF with AV-GS | RIR generation · SoundSpaces | T60 (%): 2.47 -> 2.23 | Helped |
| replace AV-NeRF with AV-GS | RIR generation · SoundSpaces | C50 (dB): 0.57 -> 0.53 | Helped |
| replace AV-NeRF with AV-GS | RIR generation · SoundSpaces | EDT (sec): 0.016 -> 0.014 | Helped |
| add audio-aware point management (ablation) | AV-GS with 3D-GS initialization · RWAVS overall | MAG: 1.481 -> 1.417 | Helped |
| add audio-aware point management (ablation) | AV-GS with SfM initialization · RWAVS overall | MAG: 1.668 -> 1.552 | Helped |
| use SH and rotation R to initialize alpha (ablation) | AV-GS · RWAVS overall | MAG: 1.435 -> 1.417 | Helped |
| use the nearest 15% of points for vicinity context (ablation) | AV-GS · RWAVS overall | MAG: 1.482 -> 1.417 | Helped |
| use the nearest 20% rather than 15% of points for vicinity context (ablation) | AV-GS · RWAVS overall | MAG: 1.417 -> 1.424 | Hurt |
| use the nearest 25% rather than 15% of points for vicinity context (ablation) | AV-GS · RWAVS overall | MAG: 1.417 -> 1.428 | Hurt |
| render 2 rather than 1 view (ablation) | AV-NeRF · RWAVS House scene 1 | MAG: 1.504 -> 1.495 | Helped |
Surprising findings
- Initializing the audio-guidance parameter from SH and rotation features works best, while adding more physical parameters (e.g., scale, opacity) often hurts performance.
Limitations
- Learns a separate representation per scene, limiting generalizability across scenes.
- Performance drops as scene size and geometric complexity increase (e.g., Office to House).
Experimental setup
Datasets: RWAVS SoundSpaces
Benchmarks: RWAVS SoundSpaces
Metrics: MAG ENV T60 C50 EDT
Baselines: Mono-Mono Mono-Energy Stereo-Energy INRAS NAF AV-NeRF Opus-nearest Opus-linear AAC-nearest AAC-linear
Builds on: 3D Gaussian Splatting AV-NeRF Neural Acoustic Field (NAF) INRAS
Closest papers by meaning
- NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields (ICLR 2025 · similarity 0.88)
- AV-Cloud: Spatial Audio Rendering Through Audio-Visual Cloud Splatting (NeurIPS 2024 · similarity 0.88)
- Acoustic Volume Rendering for Neural Impulse Response Fields (NeurIPS 2024 · similarity 0.87)
- SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding (CVPR 2025 · similarity 0.87)
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.86)
- Hearing Anything Anywhere (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.