AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Visual Acoustic Matching

Changan Chen, Ruohan Gao, Paul Calamia, Kristen Grauman

CVPR 2022 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual source localization · 45 citations (OpenAlex)

Check your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use AViTAR rather than AV U-Netvisual acoustic matching · SoundSpaces-Speech UnseenRTE (s): 0.118 -> 0.062Improved
use AViTAR rather than AV U-Netvisual acoustic matching · Acoustic AVSpeech UnseenMOSE: 0.540 -> 0.453Improved
use AViTAR rather than AV U-Netvisual acoustic matching · SoundSpaces-Speech UnseenSTFT: 0.658 -> 0.822Hurt
use image regions through cross-modal attention instead of pooled visual features (ablation)AViTAR · SoundSpaces-Speech UnseenRTE (s): 0.067 -> 0.062Improved
replace generative loss with multi-resolution STFT loss (ablation)AViTAR · SoundSpaces-Speech UnseenSTFT: 0.822 -> 0.777Improved
replace generative loss with multi-resolution STFT loss (ablation)AViTAR · SoundSpaces-Speech UnseenRTE (s): 0.062 -> 0.081Hurt
remove the visible 3D speaker (ablation)AViTAR · SoundSpaces-Speech UnseenSTFT: 0.822 -> 0.884Hurt
replace the target image with a random image (ablation)AViTAR · SoundSpaces-Speech UnseenRTE (s): 0.062 -> 0.236Hurt
add noise after dereverberation and acoustic randomization (ablation)AViTAR self-supervised training · Acoustic AVSpeech UnseenRTE (s): 0.197 -> 0.183Improved
add acoustic randomization after dereverberation (ablation)AViTAR self-supervised training · Acoustic AVSpeech UnseenRTE (s): 0.208 -> 0.183Improved
add dereverberation before randomization and noise (ablation)AViTAR self-supervised training · Acoustic AVSpeech UnseenRTE (s): 0.249 -> 0.183Improved
remove the mel-spectrogram loss (ablation)AViTAR GAN training · SoundSpaces-Speech UnseenMOSE: 0.195 -> 0.833Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy