Research map / Video / Video segmentation and tracking
Learning to Separate Object Sounds by Watching Unlabeled Video
Ruohan Gao, Rogerio Feris, Kristen Grauman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use learned object-specific audio bases instead of bases from a single exemplar video (ablation) | supervised NMF source separation · AudioSet instrument pairs | SDR (dB): -2.41 -> 1.83 | Improved |
| retrieve bases of the correct visually detected class instead of random wrong-class bases (ablation) | supervised NMF source separation · AudioSet instrument pairs | SDR (dB): -2.12 -> 1.83 | Improved |
| use learned bases instead of random Gaussian bases (ablation) | supervised NMF source separation · AudioSet instrument pairs | SDR (dB): -8.74 -> 1.83 | Improved |
| use learned bases instead of random Gaussian bases (ablation) | supervised NMF source separation · AudioSet cross-domain pairs | SDR (dB): -8.21 -> 2.53 | Improved |
| use visually guided object bases instead of K-means clustering of NMF channels (ablation) | unsupervised NMF separation · AudioSet animal pairs | SDR (dB): -3.76 -> 0.23 | Improved |
| use visually guided object bases instead of MFCC unsupervised separation | unsupervised separation · AudioSet instrument pairs | SDR (dB): 0.47 -> 1.83 | Improved |
| train with ground-truth AudioSet labels instead of ImageNet-CNN weak labels (ablation) | deep MIML network · AudioSet instrument pairs | SDR (dB): 1.83 -> 2.05 | Improved |
| use ImageNet ResNet-152 weak labels instead of ground-truth labels (ablation) | deep MIML network · AudioSet pairs | SDR (dB) | No clear effect |
| replace Pu et al. with the proposed method for denoising | AV-Bench denoising · AV-Bench denoising (Wooden Horse) | NSDR (dB): 8.82 -> 12.3 | Improved |
| replace Pu et al. with the proposed method for denoising | AV-Bench denoising · AV-Bench denoising (Guitar Solo) | NSDR (dB): 14.1 -> 11.4 | Hurt |
| replace Pu et al. with the proposed method for denoising | AV-Bench denoising · AV-Bench denoising (average) | NSDR (dB): 9.61 -> 10.5 | Improved |
Closest papers by meaning
- Co-Separating Sounds of Visual Objects (ICCV 2019 · similarity 0.94)
- Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds (ICLR 2021 · similarity 0.91)
- Language-Guided Audio-Visual Source Separation via Trimodal Consistency (CVPR 2023 · similarity 0.91)
- Learning Audio-Visual Dynamics Using Scene Graphs for Audio Source Separation (NeurIPS 2022 · similarity 0.91)
- SoundNet: Learning Sound Representations from Unlabeled Video (NeurIPS 2016 · similarity 0.90)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
