AI Paper Atlas

Research map / Video / Video segmentation and tracking

Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds

Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Dan Ellis, John R. Hershey

ICLR 2021 · Video › Video segmentation and tracking · approach: Sign language and physiological video analysis · 36 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add a small amount of human-labeled training examplesAudioScope · YFCC100m mixture of mixturesOSR (dB): 5.3 dB and 10.7 dBImproved
Use exact cross-entropy instead of active combinations or multiple-instance lossAudioScope · YFCC100m mixture of mixturesOSR (dB): 10.8; 9.4; 5.3Hurt
Add 25% synthetic off-screen examples to semi-supervised trainingAudioScope · YFCC100m mixture of mixturesOSR (dB): 10.7 -> 61.5Improved
Remove video conditioning from separation (ablation)AudioScope · YFCC100m single mixturesSI-SNR, on-screen estimate: 13.5 -> 11.5Hurt
Remove video conditioning from separation (ablation)AudioScope · YFCC100m mixture of mixturesSI-SNR, on-screen estimate: 6.3 -> 5.4Hurt
Train audio and video embedding networks from scratch instead of using pretrained weights (ablation)AudioScope · YFCC100m single mixturesAUC: 0.58 -> 0.45Hurt
Remove global video and audio embeddings from classifier input (ablation)AudioScope, unsupervised · YFCC100m single mixturesSI-SNR, on-screen estimate: 13.5 -> 14.0Mixed
Reduce attention embedding dimension from 256 to 128 (ablation)AudioScope, unsupervised · YFCC100m single mixturesSI-SNR, on-screen estimate: 13.5 -> 12.1Hurt
Replace attentional pooling for video with mean pooling (ablation)AudioScope, unsupervised · YFCC100m mixture of mixturesOSR: 9.4 -> 6.2Hurt
Use unfiltered on-screen and filtered off-screen data for unsupervised training (ablation)AudioScope · YFCC100m single mixturesAUC: 0.58 -> 0.63Improved
Use unfiltered on-screen and filtered off-screen data for unsupervised training (ablation)AudioScope · YFCC100m mixture of mixturesOSR: 9.4 -> 5.6Hurt
Increase output sources from 4 to 8 (ablation)AudioScope, unsupervised · YFCC100m mixture of mixturesSI-SNR, MixIT*: 10.5 dB -> 11.1 dBImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy