Research map / Detection and segmentation / Weakly supervised detection and segmentation
Self-Supervised Object Detection From Audio-Visual Correspondence
Triantafyllos Afouras, Yuki M. Asano, Francois Fagan, Andrea Vedaldi, Florian Metze
Check your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use audio-visual self-labels to train an image-only detector rather than relying on heatmap localization | Self-supervised object detection · Object detection and sound source localization benchmarks | mAP | Improved |
| Use ground-truth video categories instead of self-generated class labels | Ours - weak sup. · VGGSound and AudioSet | AP50: 3% AP50 drop | Improved |
| Use 20 clusters rather than 39 (ablation) | Ours · VGGSound instruments | mAP50: 39.4 -> 34.4 | Hurt |
| Use 50 clusters rather than 39 (ablation) | Ours · VGGSound instruments | mAP50: 39.4 -> 41.0 | Improved |
| Use 50 clusters rather than 39 (ablation) | Ours · OpenImages instruments | mAP50: 28.5 -> 27.5 | Hurt |
| Replace Hungarian matching with argmax matching (ablation) | Ours · VGGSound instruments | mAP50: 39.4 -> 39.6 | No clear effect |
| Replace Hungarian matching with argmax matching (ablation) | Ours · OpenImages instruments | mAP50: 28.5 -> 30.1 | Improved |
| Use manual grouping instead of Hungarian matching (ablation) | Ours · VGGSound instruments | mAP50: 39.4 -> 41.0 | Improved |
| Use manual grouping instead of Hungarian matching (ablation) | Ours · OpenImages instruments | mAP50: 28.5 -> 29.5 | Improved |
| Align clusters with only 1 labelled example per cluster (ablation) | Ours · VGGSound instruments | mAP50: 39.4 -> 36.4 | Hurt |
| Align clusters with 10 labelled examples per cluster instead of 1 (ablation) | Ours · VGGSound instruments | mAP50: 36.4 -> 37.1 | Improved |
| Sample the box-threshold parameter beta from 0.7-0.9 instead of fixing it at 0.7 (ablation) | Ours · VGGSound instruments | mAP50: 39.4 -> 37.5 | Hurt |
Closest papers by meaning
- Self-Supervised Learning of Audio-Visual Objects from Video (ECCV 2020 · similarity 0.92)
- There Is More Than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking With Sound by Distilling Multimodal Knowledge (CVPR 2021 · similarity 0.90)
- Class-Aware Sounding Objects Localization via Audiovisual Correspondence (TPAMI 2021 · similarity 0.90)
- Look, Listen and Learn (ICCV 2017 · similarity 0.89)
- Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching (NeurIPS 2020 · similarity 0.89)
- Audio-Visual Segmentation via Unlabeled Frame Exploitation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
