Research map / Detection and segmentation / Open-set and few-shot detection
There Is More Than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking With Sound by Distilling Multimodal Knowledge
Francisco Rivera Valverde, Juana Valeria Hurtado, Abhinav Valada
Check your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill from RGB, depth, and thermal teachers using MTA rather than a single RGB teacher with Ranking loss | Audio student on MAVD · MAVD test | mAP@ Avg: 44.05 -> 61.62 | Improved |
| Use multi-teacher MTA rather than averaging teacher activations | MM-DistillNet audio student · MAVD test | mAP@ Avg: 51.63 -> 61.62 | Improved |
| Use MTA rather than Ranking loss for single-teacher distillation | Audio student with RGB teacher · MAVD test | mAP@ Avg: 44.05 -> 44.58 | Improved |
| Use MTA rather than Ranking loss for single-teacher distillation | Audio student with depth teacher · MAVD test | mAP@ Avg: 45.85 -> 55.41 | Improved |
| Add thermal teacher to RGB and depth teachers (ablation) | MM-DistillNet audio student · MAVD test | mAP@ Avg: 42.89 -> 61.10 | Improved |
| Add depth teacher to RGB and thermal teachers (ablation) | MM-DistillNet audio student · MAVD test | mAP@ Avg: 55.81 -> 61.10 | Improved |
| Add car-counting pretext-task initialization to the audio student (ablation) | MM-DistillNet with three teachers · MAVD test | mAP@ Avg: 61.10 -> 61.62 | Improved |
| Initialize the audio student with car-counting pretext-task weights (ablation) | MM-DistillNet with three teachers · MAVD training | training loss: 27.55% lower | Improved |
| Use Pairwise loss instead of MTA loss | Audio student with RGB teacher · MAVD test | mAP@ Avg: 40.45 vs 44.58 | Hurt |
| Average Ranking-loss teacher activations instead of integrating them with MTA | Audio student with RGB, depth, and thermal teachers · MAVD test | mAP@ Avg: 56.16 vs 61.62 | Hurt |
| Average AFD-loss teacher activations instead of integrating them with MTA | Audio student with RGB, depth, and thermal teachers · MAVD test | mAP@ Avg: 58.50 vs 61.62 | Hurt |
| Use MTA with r=4, t=9 instead of r=2, t=9 (ablation) | Audio student with RGB teacher · MAVD test | mAP@ Avg: 69.72 -> 60.13 | Hurt |
Closest papers by meaning
- Self-Supervised Object Detection From Audio-Visual Correspondence (CVPR 2022 · similarity 0.90)
- Self-Supervised Learning of Audio-Visual Objects from Video (ECCV 2020 · similarity 0.88)
- Self-Supervised Moving Vehicle Tracking With Stereo Sound (ICCV 2019 · similarity 0.88)
- Leveraging Acoustic Images for Effective Self-Supervised Audio Representation Learning (ECCV 2020 · similarity 0.88)
- Enhanced Audio Tagging via Multi- to Single-Modal Teacher-Student Mutual Learning (AAAI 2021 · similarity 0.87)
- Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching (NeurIPS 2020 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
