Research map / Image generation / Autoregressive and latent generation
MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MelFusion, a text-to-music latent diffusion model that injects image semantics via a learned "visual synapse" built from DDIM-inverted Stable Diffusion self-attention features, and introduces the MeLBench dataset and IMSM metric.
Problem
Music generation is conditioned almost only on text, so visual cues that help composers are unused; existing text-to-music models cannot take image plus text conditioning.
Approach
Inverts the image with DDIM into a pretrained text-to-image Stable Diffusion, extracts its decoder self-attention keys/values, and mixes them into the text-to-music LDM cross-attention via learned per-layer alpha parameters (the "visual synapse").
Key results
- MelFusion achieves FAD 1.12 on MusicCaps and 1.05 on MeLBench, a relative FAD gain of up to 67.98% over MusicGen.
- Subjective OVL of 88.45 on MeLBench vs 84.61 for MusicGen and 85.52 for TANGO++.
- Conditioning on text alone or image alone performs worse than on both modalities (e.g., MusicCaps FAD 3.07 text-only, 5.62 image-only, 1.12 both).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add image conditioning through the visual synapse to text conditioning (ablation) | MELFUSION · Extended MusicCaps | FAD: 3.07 -> 1.12 | Helped |
| add text conditioning to image conditioning (ablation) | MELFUSION · Extended MusicCaps | FAD: 5.62 -> 1.12 | Helped |
| add image conditioning through the visual synapse to text conditioning (ablation) | MELFUSION · MeLBench | FAD: 3.11 -> 1.05 | Helped |
| add text conditioning to image conditioning (ablation) | MELFUSION · MeLBench | FAD: 4.16 -> 1.05 | Helped |
| place the visual synapse in the decoder rather than the encoder (ablation) | MELFUSION · Extended MusicCaps | FAD: 2.03 -> 1.12 | Helped |
| use different α parameters for each block rather than a shared α (ablation) | MELFUSION · Extended MusicCaps | FAD: 2.71 -> 1.12 | Helped |
| set α learning rate to 1e-5 rather than 0.5e-6 (ablation) | MELFUSION · Extended MusicCaps | FAD: 3.12 -> 1.12 | Helped |
| increase denoising steps from 50 to 400 (ablation) | MELFUSION · MeLBench | FAD: 2.59 -> 1.12 | Helped |
| increase classifier-free guidance from 2 to 7 (ablation) | MELFUSION · MeLBench | FAD: 1.47 -> 1.12 | Helped |
| use image conditioning with a short text prompt instead of longer text-only prompts (ablation) | MELFUSION · MeLBench | FAD: 3.02 -> 1.86 | Helped |
Surprising findings
- Image conditioning with a short text prompt (≤7 words) outperforms verbose text prompts of 20+ words without image conditioning, as shown in the verbosity study.
Limitations
- Baselines needed to be constructed because no existing model conditions on both image and text for music.
- Some baseline codebases (MuBERT, Noise2Music, MeLoDy) were not public, so they were omitted on MeLBench.
- IMSM relies on CLIP and CLAP embeddings and is validated only with a user study of 64 participants.
Experimental setup
Datasets: MeLBench MusicCaps Extended MusicCaps
Benchmarks: MusicCaps MeLBench
Metrics: FAD KL divergence FD IMSM OVL REL
Baselines: Riffusion MuBERT MusicLM Mousai Noise2Music MeLoDy MusicGen MusicLM + InstructBLIP TANGO++ AudioLDM TANGO
Builds on: Stable Diffusion TANGO AudioLDM DDIM inversion HiFi-GAN
Closest papers by meaning
- Music Style Transfer with Time-Varying Inversion of Diffusion Models (AAAI 2024 · similarity 0.88)
- MusER: Musical Element-Based Regularization for Generating Symbolic Music with Emotion (AAAI 2024 · similarity 0.88)
- MusicRL: Aligning Music Generation to Human Preferences (ICML 2024 · similarity 0.87)
- Compose with Me: Collaborative Music Inpainter for Symbolic Music Infilling (AAAI 2025 · similarity 0.87)
- Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation (ICML 2025 · similarity 0.87)
- ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.