Research map / Video / Video segmentation and tracking
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
Andrew Owens, Jiajun Wu, Josh H. McDermott, William T. Freeman, Antonio B. Torralba
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Predict full sound texture statistics instead of only the frequency spectrum (ablation) | Sound-prediction CNN (cluster) · PASCAL VOC 2007 classification (pool5) | mAP (%): 44.0 -> 45.8 | Improved |
| Predict full sound texture statistics instead of only the frequency spectrum (ablation) | Sound-prediction CNN (cluster) · SUN397 (pool5) | accuracy (%): 19.5 -> 22.9 | Improved |
| Replace audio texture features with visual texton histograms as the label space (ablation) | Cluster model · PASCAL VOC 2007 classification (pool5) | mAP (%): 45.8 -> 37.5 | Hurt |
| Replace audio texture features with visual texton histograms as the label space (ablation) | Cluster model · SUN397 (pool5) | accuracy (%): 22.9 -> 15.2 | Hurt |
| Train with human AudioSet sound annotations instead of unsupervised audio labels (ablation) | Binary-style model on AudioSet · PASCAL VOC 2007 classification (pool5) | mAP (%): 48.6 -> 55.6 | Improved |
| Increase the number of audio clusters from 30 to 120 (ablation) | Cluster model on AudioSet · PASCAL VOC 2007 classification (pool5) | mAP (%): 47.5 -> 48.8 | Improved |
| Use only two audio clusters instead of 30 (ablation) | Cluster model · PASCAL VOC 2007 classification | mAP | Hurt |
| Initialize Fast R-CNN with sound-trained weights (cluster) instead of random initialization | Fast R-CNN · PASCAL VOC 2007 detection | mAP (%): 41.3 -> 44.1 | Improved |
| Use global max pooling of conv5 (max5) features with sound-binary model instead of colorization | Sound (binary) · PASCAL VOC 2007 classification (max5) | mAP (%): 38.8 -> 39.4 | Improved |
| Compare sound-binary fc7 features to colorization fc7 features | Sound (binary) · PASCAL VOC 2007 classification (fc7) | mAP (%): 51.0 -> 47.4 | Hurt |
| Train on ambient sound instead of visual tracking supervision | Sound (cluster) · SUN397 (pool5) | accuracy (%): 18.7 -> 22.9 | Improved |
| Train on ambient sound instead of Places scene labels for sound-making object detectors | Sound (cluster) · Object-selective units | fraction of detectors with characteristic sounds: 26 -> 49 | Improved |
Closest papers by meaning
- Ambient Sound Provides Supervision for Visual Learning (ECCV 2016 · similarity 0.98)
- SoundNet: Learning Sound Representations from Unlabeled Video (NeurIPS 2016 · similarity 0.89)
- Objects that Sound (ECCV 2018 · similarity 0.89)
- Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications (TPAMI 2019 · similarity 0.88)
- Binaural SoundNet: Predicting Semantics, Depth and Motion With Binaural Sounds (TPAMI 2022 · similarity 0.88)
- Semantic Object Prediction and Spatial Sound Super-Resolution with Binaural Sounds (ECCV 2020 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
