AI Paper Atlas

Research map / Video / Video generation

Visually-Guided Audio Spatialization in Video with Geometry-Aware Multi-task Learning

Rishabh Garg, Ruohan Gao, Kristen Lorraine Grauman

IJCV 2023 · Video › Video generation · approach: Video-to-audio synthesis · 39 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add RIR prediction loss (ablation)backbone · SimBinaural Position-SplitSTFT distance: 0.805 -> 0.713Improved
add spatial coherence loss (ablation)backbone · FAIR-PlaySTFT distance: 0.927 -> 0.873Improved
add geometric consistency loss (ablation)backbone · FAIR-PlaySTFT distance: 0.927 -> 0.874Improved
combine all three auxiliary losses instead of RIR prediction alone (ablation)Backbone+IR Pred · SimBinaural Position-SplitSTFT distance: 0.713 -> 0.691Improved
combine all three losses instead of RIR prediction alone (ablation)Backbone+IR Pred · SimBinaural Scene-SplitSTFT distance: 0.801 -> 0.795Improved
add geometric consistency to RIR prediction and spatial coherence (ablation)IR Pred+Spatial · SimBinaural Position-SplitSTFT distance: 0.702 -> 0.691Improved
add RIR prediction to spatial and geometric losses (ablation)Spatial+Geometric · SimBinaural Position-SplitSTFT distance: 0.724 -> 0.691Improved
add spatial coherence to RIR prediction and geometric losses (ablation)IR Pred+Geometric · SimBinaural Position-SplitSTFT distance: 0.707 -> 0.691Improved
combine spatial coherence and geometric consistency (ablation)Backbone+Spatial · FAIR-PlaySTFT distance: 0.873 -> 0.869Improved
flip visual frames at test timeOur method · SimBinaural Position-SplitSTFT distance: 0.691 -> 1.075Hurt
add SimBinaural data to FAIR-Play trainingOurs · FAIR-Play (extra data)STFT distance: 1.234 -> 1.175Improved
remove visual input (audio-only)Our method · SimBinaural Position-SplitSTFT distance: 0.691 -> 0.932Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy