Research map / Vision-language and multimodal / Cross-modal learning
Objects that Sound
Relja Arandjelovic, Andrew Zisserman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compute correspondence from Euclidean distance between aligned embeddings (AVE-Net) instead of concatenation and FC layers (L3-Net) | L3-Net · AudioSet-Instruments cross-modal retrieval (image-to-audio) | nDCG@30: .418 -> .561 | Improved |
| same change as above | L3-Net · AudioSet-Instruments cross-modal retrieval (audio-to-image) | nDCG@30: .385 -> .587 | Improved |
| same change as above | L3-Net · AudioSet-Instruments intra-modal retrieval (audio-to-audio) | nDCG@30: .653 -> .665 | Improved |
| train AVE-Net from unlabelled video instead of using supervised ImageNet VGG16 features | VGG16-ImageNet · AudioSet-Instruments image-to-image | nDCG@30: .600 -> .604 | Improved |
| align L3-Net features with CCA | L3-Net · AudioSet-Instruments image-to-audio | nDCG@30: .418 -> .531 | Improved |
| align ImageNet VGG16 vision features with L3 audio features by CCA instead of learning embeddings | AVE-Net · AudioSet-Instruments image-to-audio | nDCG@30: .561 -> .458 | Hurt |
| use multi-frame (25 frames) visual input (ablation) | AVE-Net · AudioSet-Instruments AVC | AVC accuracy: 81.9% -> 84.7% | Improved |
| use frame plus optical flow visual input (ablation) | AVE-Net · AudioSet-Instruments AVC | AVC accuracy: 81.9% -> 84.9% | Improved |
| use frame plus optical flow visual input (ablation) | AVE-Net · AudioSet-Instruments image-to-image retrieval | nDCG@30: .604 -> .608 | No clear effect |
| use frame plus optical flow visual input (ablation) | AVE-Net · AudioSet-Instruments image-to-audio retrieval | nDCG@30: .561 -> .558 | No clear effect |
| add shortcut prevention by sampling negative audio only at multiples of 0.04s (ablation) | AVE-Net · AudioSet-Instruments retrieval | nDCG@30: 1-2% better retrieval | Improved |
| add shortcut prevention by sampling negative audio only at multiples of 0.04s (ablation) | AVE-Net · AudioSet-Instruments AVC | AVC accuracy: 87.6% -> 81.9% | Mixed |
Closest papers by meaning
- Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications (TPAMI 2019 · similarity 0.90)
- Improving Sound Source Localization with Joint Slot Attention on Image and Audio (CVPR 2025 · similarity 0.89)
- Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language (CVPR 2024 · similarity 0.89)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.88)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.88)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
