Research map / Image generation / Text-to-image diffusion
Images that Sound: Composing Images and Sounds on a Single Canvas
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Zero-shot composition of pretrained text-to-image (Stable Diffusion v1.5) and text-to-spectrogram (Auffusion) latent diffusion models produces spectrograms that look like images and sound like audio prompts.
Problem
Generating spectrograms that simultaneously look like natural images and sound like natural audio, given no paired image-spectrogram data.
Approach
Jointly denoise a shared latent space by summing classifier-free-guided noise estimates from an image diffusion model and an audio (spectrogram) diffusion model, sampling from the intersection of both distributions, with warm-starting and optional colorization.
Key results
- Ours achieves CLIP 28.2% and CLAP 33.5% versus imprint 27.2% / 32.3% and SDS 25.4% / 23.4%, with best FID 226.46 and FAD 19.21.
- Human study win-rates against baselines average 77.4% (audio quality) and 73.0% (alignment) versus SDS, and 74.4% and 78.9% versus imprint.
- Generating one sample takes about 10 seconds on an NVIDIA L40s versus about 2 hours for the SDS baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compose image and spectrogram diffusion denoising instead of single-modality generation | Ours · 100 prompt pairs | CLIP, CLAP, FID, FAD: CLIP: 27.2 -> 28.2; CLAP: 32.3 -> 33.5; FID: 244.84 -> 226.46; FAD: 29.42 -> 19.21 | Helped |
| Use multimodal denoising instead of SDS optimization | Ours vs SDS · 100 prompt pairs | CLIP, CLAP, FID, FAD: CLIP: 25.4 -> 28.2; CLAP: 23.4 -> 33.5; FID: 273.03 -> 226.46; FAD: 32.57 -> 19.21 | Helped |
| Use HiFi-GAN instead of Griffin-Lim (ablation) | Ours · Vocoder ablation | CLAP: 0.302 -> 0.335 | Helped |
| Increase both guidance scales from 5.0 to 10 (ablation) | Ours · Guidance scale | CLAP (%): 29.3 -> 33.5 | Helped |
| Increase both guidance scales from 5.0 to 10 (ablation) | Ours · Guidance scale | CLIP (%): 28.0 -> 28.2 | Helped |
| Warm-start with audio denoising alone for the first 10% of timesteps (tv=0.9, ta=1.0) (ablation) | Ours · Warm-start | CLAP (%): 30.8 -> 33.5 | Helped |
| Warm-start with audio denoising alone for the first 10% of timesteps (tv=0.9, ta=1.0) (ablation) | Ours · Warm-start | Overall: 1.14 -> 1.15 | Helped |
| Warm-start with audio denoising alone (tv=1.0, ta=0.9) (ablation) | Ours · Warm-start | CLAP (%): 30.8 -> 27.4 | Hurt |
| Remove the grayscale style word (ablation) | Ours · Style word ablation | CLIP: 28.2 -> 28.1 | No clear effect |
| Remove the grayscale style word (ablation) | Ours · Style word ablation | CLAP: 33.5 -> 33.7 | Helped |
| Remove the grayscale style word (ablation) | Ours · Style word ablation | FID: 226.46 -> 237.12 | Hurt |
| Remove the grayscale style word (ablation) | Ours · Style word ablation | FAD: 19.21 -> 18.09 | Helped |
Surprising findings
- Off-the-shelf image and audio diffusion models trained on different modalities can be composed in a shared latent space to yield samples that function as both image and sound.
- Stable Diffusion images alone achieve a very low CLAP score (2.2%), indicating randomly sampled natural images make poor spectrograms.
Limitations
- Cannot generate examples with both high-fidelity audio and image; many prompt pairs fail.
- Quality bounded by the audio diffusion model, which lags behind visual models.
- Discrete sounds (e.g., dog barking) are harder due to sparse spectrogram energy.
Experimental setup
Datasets: VGGSound Common
Benchmarks: VGGSound Common
Metrics: CLIP score CLAP score FID FAD human win-rate
Baselines: SDS imprint Stable Diffusion Auffusion Griffin-Lim
Builds on: Stable Diffusion Auffusion Visual Anagrams Diffusion Illusions classifier-free guidance
Closest papers by meaning
- SoundBrush: Sound as a Brush for Visual Scene Editing (AAAI 2025 · similarity 0.85)
- Sounding that Object: Interactive Object-Aware Image to Audio Generation (ICML 2025 · similarity 0.85)
- Animate and Sound an Image (CVPR 2025 · similarity 0.84)
- V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · similarity 0.84)
- Customized Condition Controllable Generation for Video Soundtrack (CVPR 2025 · similarity 0.84)
- MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.