Research map / Video / Video generation
Visually-Guided Audio Spatialization in Video with Geometry-Aware Multi-task Learning
Rishabh Garg, Ruohan Gao, Kristen Lorraine Grauman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RIR prediction loss (ablation) | backbone · SimBinaural Position-Split | STFT distance: 0.805 -> 0.713 | Improved |
| add spatial coherence loss (ablation) | backbone · FAIR-Play | STFT distance: 0.927 -> 0.873 | Improved |
| add geometric consistency loss (ablation) | backbone · FAIR-Play | STFT distance: 0.927 -> 0.874 | Improved |
| combine all three auxiliary losses instead of RIR prediction alone (ablation) | Backbone+IR Pred · SimBinaural Position-Split | STFT distance: 0.713 -> 0.691 | Improved |
| combine all three losses instead of RIR prediction alone (ablation) | Backbone+IR Pred · SimBinaural Scene-Split | STFT distance: 0.801 -> 0.795 | Improved |
| add geometric consistency to RIR prediction and spatial coherence (ablation) | IR Pred+Spatial · SimBinaural Position-Split | STFT distance: 0.702 -> 0.691 | Improved |
| add RIR prediction to spatial and geometric losses (ablation) | Spatial+Geometric · SimBinaural Position-Split | STFT distance: 0.724 -> 0.691 | Improved |
| add spatial coherence to RIR prediction and geometric losses (ablation) | IR Pred+Geometric · SimBinaural Position-Split | STFT distance: 0.707 -> 0.691 | Improved |
| combine spatial coherence and geometric consistency (ablation) | Backbone+Spatial · FAIR-Play | STFT distance: 0.873 -> 0.869 | Improved |
| flip visual frames at test time | Our method · SimBinaural Position-Split | STFT distance: 0.691 -> 1.075 | Hurt |
| add SimBinaural data to FAIR-Play training | Ours · FAIR-Play (extra data) | STFT distance: 1.234 -> 1.175 | Improved |
| remove visual input (audio-only) | Our method · SimBinaural Position-Split | STFT distance: 0.691 -> 0.932 | Hurt |
Closest papers by meaning
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.91)
- Cyclic Learning for Binaural Audio Generation and Localization (CVPR 2024 · similarity 0.90)
- Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos (CVPR 2024 · similarity 0.89)
- MRSAudio: A Large-Scale Multimodal Recorded Spatial Audio Dataset with Refined Annotations (NeurIPS 2025 · similarity 0.89)
- Supervising Sound Localization by In-the-wild Egomotion (CVPR 2025 · similarity 0.89)
- Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
