Research map / Video / Video segmentation and tracking
Ambient Sound Provides Supervision for Visual Learning
Andrew Owens, Jiajun Wu, Josh H. McDermott, William T. Freeman, Antonio Torralba
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| predict sound texture statistics instead of only the instantaneous frequency spectrum (ablation) | Sound (binary) · PASCAL VOC 2007 classification (pool5) | mAP (%): 44.0 -> 46.7 | Improved |
| use sound labels instead of visual texton histogram labels (ablation) | Sound (cluster) · PASCAL VOC 2007 classification (pool5) | mAP (%): 37.5 -> 45.8 | Improved |
| use sound labels instead of visual texton histogram labels (ablation) | Sound (cluster) · SUN397 (pool5) | accuracy (%): 15.2 -> 22.9 | Improved |
| use binary coding labels instead of clustering labels (ablation) | Sound · PASCAL VOC 2007 classification (pool5) fc7 | mAP (%): 44.3 -> 47.4 | Improved |
| increase number of audio clusters (ablation) | Sound (cluster) · PASCAL VOC 2007 classification (pool5) | mAP (%) | Improved |
| use global max pooling of conv5 features (max5) in place of other layers (ablation) | Sound (binary) · PASCAL VOC 2007 classification | mAP (%): 39.4 vs 46.7 (pool5) | Mixed |
| add sound-pretrained initialization before Fast R-CNN finetuning (ablation) | Sound (cluster) · PASCAL VOC 2007 detection | mAP (%): 41.3 -> 44.1 | Improved |
| use sound supervision instead of tracking supervision (ablation) | Sound (cluster) · PASCAL VOC 2007 detection | mAP (%): 44.1 vs 47.4 | No clear effect |
| use sound supervision instead of egomotion supervision | Sound (cluster) · PASCAL VOC 2007 classification (pool5) | mAP (%): 31.1 -> 45.8 | Improved |
| train on ambient sound instead of human scene labels | Sound (cluster) · Object-selective units | number of detectors for objects with characteristic sounds: 26 -> 49 | Improved |
Closest papers by meaning
- Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning (IJCV 2018 · similarity 0.98)
- SoundNet: Learning Sound Representations from Unlabeled Video (NeurIPS 2016 · similarity 0.89)
- Objects that Sound (ECCV 2018 · similarity 0.87)
- Self-Supervised Audio-Visual Soundscape Stylization (ECCV 2024 · similarity 0.87)
- Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications (TPAMI 2019 · similarity 0.87)
- SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
