Research map / Detection and segmentation / Open-set and few-shot detection
Clink! Chop! Thud! - Learning Object Sounds from Real-World Interactions
Mengyu Yang, Yiming Chen, Haozheng Pei, Siddhant Agarwal, Arun Balajee Vasudevan, James Hays
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add object-aware masking and slot attention encoder to SoundingActions | SoundingActions · Sounding object detection, Epic Kitchens | top-1 accuracy (%): 41.0 -> 49.6 | Improved |
| add object-aware masking on top of slot attention (ablation) | Ours · Sounding object detection, Ego4D | top-1 accuracy (%): 40.4 -> 43.9 | Improved |
| add slot attention encoder on top of object masking (ablation) | Ours · Sounding object detection, Epic Kitchens | top-1 accuracy (%): 46.1 -> 49.6 | Improved |
| use all sequence permutations of slot attention at evaluation (ablation) | Ours · Sounding object detection, Epic Kitchens | top-1 accuracy (%): 49.6 -> 52.8 | Improved |
| add object-aware training (masks and slot attention) to MC3 pretraining | SoundingActions · Sounding action discovery, audio-vision | AUC-ROC: 0.592 -> 0.617 | Improved |
| apply object masking without slot attention in action discovery (ablation) | SoundingActions · Sounding action discovery, audio-vision | AUC-ROC: 0.592 -> 0.578 | Hurt |
| remove object masking while keeping slot attention (ablation) | Ours · Sounding action discovery, audio-vision | AUC-ROC: 0.592 -> 0.593 | No clear effect |
| use all sequence permutations (Ours++) for action discovery (ablation) | Ours · Sounding action discovery | AUC-ROC | No clear effect |
| finetune SLAVC on the task data | SLAVC · Sounding object detection, Ego4D | top-1 accuracy (%): 33.0 -> 31.5 (Ego4D), 34.8 -> 35.8 (Epic Kitchens) | Mixed |
Closest papers by meaning
- Sounding that Object: Interactive Object-Aware Image to Audio Generation (ICML 2025 · similarity 0.87)
- Visually Indicated Sounds (CVPR 2016 · similarity 0.87)
- SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos (CVPR 2024 · similarity 0.87)
- Sound Bridge: Associating Egocentric and Exocentric Videos via Audio Cues (CVPR 2025 · similarity 0.86)
- EgoSound: Benchmarking Sound Understanding in Egocentric Videos (CVPR 2026 · similarity 0.85)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
