AI Paper Atlas

Research map / Video / Video segmentation and tracking

Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning

Andrew Owens, Jiajun Wu, Josh H. McDermott, William T. Freeman, Antonio B. Torralba

IJCV 2018 · Video › Video segmentation and tracking · approach: Sign language and physiological video analysis · 62 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Predict full sound texture statistics instead of only the frequency spectrum (ablation)Sound-prediction CNN (cluster) · PASCAL VOC 2007 classification (pool5)mAP (%): 44.0 -> 45.8Improved
Predict full sound texture statistics instead of only the frequency spectrum (ablation)Sound-prediction CNN (cluster) · SUN397 (pool5)accuracy (%): 19.5 -> 22.9Improved
Replace audio texture features with visual texton histograms as the label space (ablation)Cluster model · PASCAL VOC 2007 classification (pool5)mAP (%): 45.8 -> 37.5Hurt
Replace audio texture features with visual texton histograms as the label space (ablation)Cluster model · SUN397 (pool5)accuracy (%): 22.9 -> 15.2Hurt
Train with human AudioSet sound annotations instead of unsupervised audio labels (ablation)Binary-style model on AudioSet · PASCAL VOC 2007 classification (pool5)mAP (%): 48.6 -> 55.6Improved
Increase the number of audio clusters from 30 to 120 (ablation)Cluster model on AudioSet · PASCAL VOC 2007 classification (pool5)mAP (%): 47.5 -> 48.8Improved
Use only two audio clusters instead of 30 (ablation)Cluster model · PASCAL VOC 2007 classificationmAPHurt
Initialize Fast R-CNN with sound-trained weights (cluster) instead of random initializationFast R-CNN · PASCAL VOC 2007 detectionmAP (%): 41.3 -> 44.1Improved
Use global max pooling of conv5 (max5) features with sound-binary model instead of colorizationSound (binary) · PASCAL VOC 2007 classification (max5)mAP (%): 38.8 -> 39.4Improved
Compare sound-binary fc7 features to colorization fc7 featuresSound (binary) · PASCAL VOC 2007 classification (fc7)mAP (%): 51.0 -> 47.4Hurt
Train on ambient sound instead of visual tracking supervisionSound (cluster) · SUN397 (pool5)accuracy (%): 18.7 -> 22.9Improved
Train on ambient sound instead of Places scene labels for sound-making object detectorsSound (cluster) · Object-selective unitsfraction of detectors with characteristic sounds: 26 -> 49Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy