Research map / Vision-language and multimodal / Cross-modal learning
Leveraging Acoustic Images for Effective Self-Supervised Audio Representation Learning
Valentina Sanguineti, Pietro Morerio, Niccolò Pozzetti, Danilo Greco, Marco Cristani, Vittorio Murino
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add acoustic-image (DualCamNet) transfer, alpha 0.5 (ablation) | HearNet self-supervised · ACIVW test (KNN classification) | accuracy: 0.7573 -> 0.7946 | Improved |
| add acoustic-image transfer, alpha 0.9 (ablation) | Audio-visual (H) self-supervised · ACIVW test (KNN classification) | accuracy: 0.6670 -> 0.7322 | Improved |
| add acoustic-image transfer, alpha 0.7 (ablation) | HearNet self-supervised · AVIA (trained on ACIVW, KNN) | accuracy: 0.4103 -> 0.4851 | Improved |
| add acoustic-image transfer, alpha 0.3 (ablation) | Audio-visual (H) self-supervised · AVIA (trained on ACIVW, KNN) | accuracy: 0.2660 -> 0.3200 | Improved |
| add acoustic-image transfer, alpha 0.3 (ablation) | HearNet self-supervised · DCASE 2018 (trained on ACIVW, KNN) | accuracy: 0.3535 -> 0.3757 | Improved |
| use self-supervised instead of supervised training (ablation) | HearNet · AVIA (trained on ACIVW, KNN) | accuracy: 0.3108 -> 0.4103 | Improved |
| use self-supervised instead of supervised training (ablation) | HearNet · DCASE 2018 (trained on ACIVW, KNN) | accuracy: 0.3022 -> 0.3535 | Improved |
| use self-supervised instead of supervised training (ablation) | HearNet · ACIVW test (KNN classification) | accuracy: 0.8779 -> 0.7573 | Hurt |
| add supervised acoustic-image distillation (ablation) | HearNet supervised · ACIVW test (KNN classification) | accuracy: 0.8779 -> 0.8578 | Hurt |
| add supervised acoustic-image distillation (ablation) | HearNet supervised · AVIA (trained on ACIVW, KNN) | accuracy: 0.3108 -> 0.3556 | Improved |
| add supervised acoustic-image distillation (ablation) | HearNet supervised · DCASE 2018 (trained on ACIVW, KNN) | accuracy: 0.3022 -> 0.2989 | No clear effect |
| use acoustic image instead of monaural audio input (DualCamNet vs HearNet) | audio-visual retrieval · ACIVW cross-modal retrieval (audio to video) | Rank 1 CMC: 28.95 -> 33.41 | Improved |
Closest papers by meaning
- Self-Supervised Learning of Audio-Visual Objects from Video (ECCV 2020 · similarity 0.88)
- Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (ECCV 2018 · similarity 0.88)
- Self-Supervised Visual Acoustic Matching (NeurIPS 2023 · similarity 0.88)
- There Is More Than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking With Sound by Distilling Multimodal Knowledge (CVPR 2021 · similarity 0.88)
- Telling Left From Right: Learning Spatial Correspondence of Sight and Sound (CVPR 2020 · similarity 0.87)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
