Research map / Detection and segmentation / Open-set and few-shot detection
Sounding that Object: Interactive Object-Aware Image to Audio Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an interactive object-aware image-to-audio diffusion model that uses multi-modal dot-product attention between CLAP text and CLIP patch embeddings during training, then swaps in SAM segmentation masks at test time for user-selected object sound generation.
Problem
Existing image- and text-to-audio models generate holistic soundscapes that miss or conflate sounds of individual objects in complex multi-source scenes, and lack object-level user control.
Approach
Fine-tune a conditional latent diffusion model (AudioLDM) with a CLAP-text-to-CLIP-patch dot-product attention that grounds sounds in image regions, then at test time replace the learned attention with SAM masks (clicks or text prompts); a theorem bounds the resulting test error via contrastive-loss alignment with dot-product attention.
Key results
- On AudioCaps, the model reaches ACC 0.859 versus 0.806 for the next best (SSV2A), with KL 0.517 and IS 2.102 also best among compared methods, while FAD 1.271 is roughly tied with SSV2A's 1.265.
- Human study (50 raters, 100 samples) gives REO 3.74 versus 3.48 for SSV2A, the strongest object-relevance margin reported.
- Replacing SAM with SAM 2 at test time improves ACC from 0.859 to 0.881 and FAD from 1.271 to 1.153.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune the diffusion weights rather than freeze them (ablation) | Ours · AudioCaps | ACC: 0.692 -> 0.859 | Helped |
| replace single-head with multi-head attention (ablation) | Ours · AudioCaps | ACC: 0.859 -> 0.415 | Hurt |
| replace dot-product attention with additive attention (ablation) | Ours · AudioCaps | ACC: 0.859 -> 0.103 | Hurt |
| use text-image attention rather than segmentation masks during inference (ablation) | Ours · AudioCaps | ACC: 0.859 -> 0.856 | No clear effect |
| replace text-image attention with audio-image attention (ablation) | Ours · AudioCaps | ACC: 0.859 -> 0.634 | Hurt |
| use segmentation masks instead of attention during training (ablation) | Ours · AudioCaps | ACC: 0.859 -> 0.763 | Hurt |
| add positional encoding (ablation) | Ours · AudioCaps | ACC: 0.787 -> 0.859 | Helped |
| increase CFG scale from 1.0 to 2.0 (ablation) | Ours · AudioCaps | ACC: 0.413 -> 0.859 | Helped |
| increase CFG scale from 2.0 to 2.5 (ablation) | Ours · AudioCaps | ACC: 0.859 -> 0.807 | Hurt |
| use a 0.6 audio-visual matching threshold rather than 0.4 (ablation) | Ours · AudioCaps | ACC: 0.521 -> 0.859 | Helped |
| caption objects separately and mix their generated clips instead of using the full scene (ablation) | Captioning · AudioCaps | ACC: 0.587 -> 0.643 | Helped |
| caption objects separately and mix their generated clips instead of using the full scene (ablation) | Captioning · AudioCaps | FAD: 2.778 -> 7.634 | Hurt |
Surprising findings
- Additive attention collapses (ACC 0.103), and using hard segmentation masks during training degrades performance (ACC 0.763 vs 0.859), which the authors attribute to overly rigid masks.
- Multi-head attention improves text-audio alignment in the ablation but reduces mask controllability (ACC 0.415).
Limitations
- Relies on static images, so it cannot produce non-stationary, time-synchronized sounds such as impacts.
- Lacks precise control over sound type for similar objects (e.g., a car making siren vs engine noise).
- Theoretical guarantee depends on assumptions (Lipschitz, bounded values) and error terms that are not empirically measured.
Experimental setup
Datasets: AudioSet AudioCaps VGG-Sound Sync ImageHear Places Greatest Hits
Benchmarks: AudioCaps VGG-Sound Sync ImageHear
Metrics: ACC FAD KL IS AVC OVL RET REI REO ImageBind score
Baselines: Retrieve & Separate AudioLDM 1 AudioLDM 2 Captioning Make-an-Audio Im2Wav SpecVQGAN Diff-Foley CoDi Seeing & Hearing FoleyCrafter SSV2A
Builds on: AudioLDM CLIP CLAP Segment Anything classifier-free guidance
Closest papers by meaning
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.88)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.88)
- SoundBrush: Sound as a Brush for Visual Scene Editing (AAAI 2025 · similarity 0.88)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.88)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.88)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.