Research map / Detection and segmentation / Open-set and few-shot detection
Generative Modeling of Audible Shapes for Object Perception
Zhoutong Zhang, Jiajun Wu, Qiujia Li, Zhengjia Huang, James Traer, Josh H. McDermott, Joshua B. Tenenbaum, William T. Freeman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the audio stream to a visual-only network for material recognition (ablation) | ResNet-18 visual stream · Material classification (Sound-20K, texture-less) | accuracy (%): 46.2 -> 68.5 | Improved |
| add the visual stream to an audio-only network for material recognition (ablation) | SoundNet-8 audio stream · Material classification (Sound-20K, texture-less) | accuracy (%): 65.6 -> 68.5 | Improved |
| add the audio stream to a visual-only network for material recognition (ablation) | ResNet-18 visual stream · Material classification (Physics 101) | accuracy (%): 94.5 -> 99.4 | Improved |
| add the visual stream to an audio-only network for material recognition (ablation) | SoundNet-8 audio stream · Material classification (Physics 101) | accuracy (%): 78.1 -> 99.4 | Improved |
| add audio to visual input for shape attribute recognition (ablation) | ResNet-18 visual stream · Shape attribute recognition (Sound-20K) | average accuracy (%): 79.4 -> 83.9 | Improved |
| add visual to audio input for shape attribute recognition (ablation) | SoundNet-8 audio stream · Shape attribute recognition (Sound-20K) | average accuracy (%): 76.1 -> 83.9 | Improved |
| add audio to visual input for shape attribute recognition (ablation) | ResNet-18 visual stream · Shape attribute recognition (Physics 101) | average accuracy (%): 96.6 -> 97.0 | Improved |
| add visual to audio input for shape attribute recognition (ablation) | SoundNet-8 audio stream · Shape attribute recognition (Physics 101) | average accuracy (%): 86.5 -> 97.0 | Improved |
| use deeper conv7 features instead of pool5 for transfer of synthetic-trained audio features (ablation) | SoundNet-8 trained on Sound-20K · Material classification transfer (Physics 101, linear SVM) | accuracy (%): 29.6 -> 41.5 | Improved |
| use deeper conv7 features instead of pool5 for transfer of synthetic-trained audio features (ablation) | SoundNet-8 trained on Sound-20K · Material classification transfer (Greatest Hits, linear SVM) | accuracy (%): 33.7 -> 39.2 | Improved |
| use synthetic-trained conv7 features instead of MFCC | linear SVM material classifier · Material classification transfer (Physics 101, linear SVM) | accuracy (%): 40.7 -> 41.5 | No clear effect |
| use synthetic-trained conv7 features instead of MFCC | linear SVM material classifier · Material classification transfer (Greatest Hits, linear SVM) | accuracy (%): 38.1 -> 39.2 | No clear effect |
Closest papers by meaning
- Sounding that Object: Interactive Object-Aware Image to Audio Generation (ICML 2025 · similarity 0.89)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.88)
- V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · similarity 0.88)
- Audible World Models: Spatially Aware Sound Generation for 3D Worlds (NeurIPS 2026 · similarity 0.87)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.86)
- Generative Perception of Shape and Material from Differential Motion (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
