Research map / 3D vision / 3D generation
AV-Cloud: Spatial Audio Rendering Through Audio-Visual Cloud Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AV-Cloud, a point-based audio-visual rendering framework that decodes sparse audio-visual anchors (from SfM points) via attention-based splatting into two masks that turn mono sound into viewpoint-specific stereo audio, without needing rendered images.
Problem
Rendering high-quality, viewpoint-dependent spatial audio for 3D scenes reconstructed from in-the-wild videos, synchronized with visual rendering and without relying on pre-rendered images (which introduce artifacts and audio lag).
Approach
Represent the scene with clustered Audio-Visual Anchors (3D coordinates, RGB features, latent audio embeddings) and use a Visual-to-Audio Splatting Transformer conditioned on listener head pose to weight anchors per frequency band; a Spatial Audio Render Head then applies predicted mixture and difference masks (with time filters and Conv2D refinement) to the mono input.
Key results
- On RWAVS, AV-Cloud reaches MAG 0.351, LRE 0.936, DPAM 0.276, outperforming the best prior baselines on audio metrics while using 3.91M parameters at 83 FPS.
- On Replay-NVAS, AV-Cloud reduces RTE to 0.065 versus 0.137 for ViGAS, a 53% relative reduction, and reaches MAG 0.180.
- In a human study with 76 participants on in-the-wild fountain routes, AV-Cloud was preferred in 48% of votes versus 37% for AVNeRF and 15% for NAF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AV-Cloud instead of the strongest listed baseline | spatial audio rendering · RWAVS validation | MAG: 0.370 -> 0.351 | Helped |
| use AV-Cloud instead of the strongest listed baseline | spatial audio rendering · RWAVS validation | LRE: 0.996 -> 0.936 | Helped |
| use AV-Cloud instead of the strongest listed baseline | spatial audio rendering · RWAVS validation | DPAM: 0.353 -> 0.276 | Helped |
| use AV-Cloud instead of the strongest listed baseline | spatial audio rendering · Replay-NVAS test | RTE: 0.137 -> 0.065 | Helped |
| replace pose-conditioned MLP baseline with AV-Cloud (ablation) | spatial audio rendering · RWAVS validation | MAG: 0.488 -> 0.351 | Helped |
| add AVCS rather than predicting masks directly from viewpoint pose (ablation) | AV-Cloud · RWAVS validation | LRE: 1.124 -> 0.936 | Helped |
| add Time Filters (ablation) | AV-Cloud · RWAVS validation | RTE: 0.084 -> 0.074 | Helped |
| add latent audio embeddings (ablation) | AV-Cloud · RWAVS validation | LRE: 0.975 -> 0.936 | Helped |
| add RGB visual features (ablation) | AV-Cloud · RWAVS validation | MAG: 0.361 -> 0.351 | Helped |
| use separate mixture and difference masks instead of a direct two-channel transfer mask (ablation) | AV-Cloud · RWAVS validation | LRE: 0.983 -> 0.936 | Helped |
| add Conv2D layers (ablation) | AV-Cloud · RWAVS validation | MAG: 0.359 -> 0.351 | Helped |
| use SH-based Time Filters instead of the full Time Filters (ablation) | AV-Cloud · RWAVS validation | RTE: 0.074 -> 0.073 | Mixed |
Surprising findings
- The smaller sim-SH variant (0.51M parameters) runs at 301 FPS on RWAVS, faster than the full model while staying competitive on accuracy.
- Increasing the anchor count to 512 slightly degraded most metrics, indicating more anchors do not necessarily help.
Limitations
- Depends on camera calibration/SfM quality; poor visual data or inconsistent calibration lowers audio quality.
- Not robust to noisy real-world recordings; ambient noise in ground truth can degrade output.
- No error bars or statistical significance reported due to compute cost and fixed seed.
Experimental setup
Datasets: RWAVS Replay-NVAS
Benchmarks: RWAVS Replay-NVAS
Metrics: MAG LRE ENV RTE DPAM FPS
Baselines: Mono-Mono Mono-Energy Stereo-Energy DSP VAM ViGAS AV-NeRF NACF INRAS NAF
Builds on: 3D Gaussian Splatting COLMAP NAF INRAS AV-NeRF
Closest papers by meaning
- AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis (NeurIPS 2024 · similarity 0.88)
- Audible World Models: Spatially Aware Sound Generation for 3D Worlds (NeurIPS 2026 · similarity 0.87)
- SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding (CVPR 2025 · similarity 0.86)
- Hearing Anything Anywhere (CVPR 2024 · similarity 0.85)
- OmniAudio: Generating Spatial Audio from 360-Degree Video (ICML 2025 · similarity 0.85)
- Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.