Research map / Vision-language and multimodal / Cross-modal learning
Visual Acoustic Matching
Changan Chen, Ruohan Gao, Paul Calamia, Kristen Grauman
Check your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AViTAR rather than AV U-Net | visual acoustic matching · SoundSpaces-Speech Unseen | RTE (s): 0.118 -> 0.062 | Improved |
| use AViTAR rather than AV U-Net | visual acoustic matching · Acoustic AVSpeech Unseen | MOSE: 0.540 -> 0.453 | Improved |
| use AViTAR rather than AV U-Net | visual acoustic matching · SoundSpaces-Speech Unseen | STFT: 0.658 -> 0.822 | Hurt |
| use image regions through cross-modal attention instead of pooled visual features (ablation) | AViTAR · SoundSpaces-Speech Unseen | RTE (s): 0.067 -> 0.062 | Improved |
| replace generative loss with multi-resolution STFT loss (ablation) | AViTAR · SoundSpaces-Speech Unseen | STFT: 0.822 -> 0.777 | Improved |
| replace generative loss with multi-resolution STFT loss (ablation) | AViTAR · SoundSpaces-Speech Unseen | RTE (s): 0.062 -> 0.081 | Hurt |
| remove the visible 3D speaker (ablation) | AViTAR · SoundSpaces-Speech Unseen | STFT: 0.822 -> 0.884 | Hurt |
| replace the target image with a random image (ablation) | AViTAR · SoundSpaces-Speech Unseen | RTE (s): 0.062 -> 0.236 | Hurt |
| add noise after dereverberation and acoustic randomization (ablation) | AViTAR self-supervised training · Acoustic AVSpeech Unseen | RTE (s): 0.197 -> 0.183 | Improved |
| add acoustic randomization after dereverberation (ablation) | AViTAR self-supervised training · Acoustic AVSpeech Unseen | RTE (s): 0.208 -> 0.183 | Improved |
| add dereverberation before randomization and noise (ablation) | AViTAR self-supervised training · Acoustic AVSpeech Unseen | RTE (s): 0.249 -> 0.183 | Improved |
| remove the mel-spectrogram loss (ablation) | AViTAR GAN training · SoundSpaces-Speech Unseen | MOSE: 0.195 -> 0.833 | Hurt |
Closest papers by meaning
- Self-Supervised Visual Acoustic Matching (NeurIPS 2023 · similarity 0.94)
- Self-Supervised Audio-Visual Soundscape Stylization (ECCV 2024 · similarity 0.89)
- Self-Supervised Learning of Audio-Visual Objects from Video (ECCV 2020 · similarity 0.88)
- Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment (CVPR 2023 · similarity 0.88)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.88)
- From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
