Research map / Image generation / Editing and style transfer
Music Style Transfer with Time-Varying Inversion of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a music style transfer method that learns a time-varying textual inversion pseudo-word from a five-second style clip and applies bias-reduced stylization on Riffusion (Stable Diffusion) mel-spectrograms.
Problem
Music style transfer from text is hard due to scarce audio-text data and coarse text descriptions; existing one-shot methods struggle with natural sounds and niche instruments.
Approach
A time-varying textual inversion module (TVE) makes the pseudo-word embedding depend on diffusion timestep, so small timesteps capture texture (style) and large ones structure; bias-reduced stylization uses partial diffusion of the content mel-spectrogram with predicted noise instead of random noise to preserve content.
Key results
- Objective content preservation 0.4645 and style fit 0.2816, the best among R+TI, SS VQ-VAE and MUSICGEN.
- User study overall quality 3.66 vs 2.83 for MUSICGEN and 2.75 for R+TI.
- Ablation: removing TVE drops style fit from 0.2816 to 0.2418; removing bias-reduced stylization drops content preservation from 0.4645 to 0.4415.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed time-varying textual inversion and bias-reduced stylization instead of R+TI | Riffusion-based music style transfer · Music style transfer | content preservation (objective CLAP): 0.3481 -> 0.4645 | Helped |
| Use the proposed method instead of R+TI | Riffusion-based music style transfer · Music style transfer | style fit (objective CLAP): 0.2722 -> 0.2816 | Helped |
| Use the proposed method instead of SS VQ-VAE | Music style transfer · Music style transfer | content preservation (objective CLAP): 0.2351 -> 0.4645 | Helped |
| Use the proposed method instead of SS VQ-VAE | Music style transfer · Music style transfer | style fit (objective CLAP): 0.2809 -> 0.2816 | Helped |
| Use the proposed method instead of MUSICGEN | Music style transfer · Music style transfer | content preservation (objective CLAP): 0.2808 -> 0.4645 | Helped |
| Use the proposed method instead of MUSICGEN | Music style transfer · Music style transfer | style fit (objective CLAP): 0.2370 -> 0.2816 | Helped |
| Add time-varying embedding (TVE) (ablation) | Riffusion-based music style transfer · Music style transfer | content preservation: 0.4506 -> 0.4645 | No clear effect |
| Add time-varying embedding (TVE) (ablation) | Riffusion-based music style transfer · Music style transfer | style fit: 0.2418 -> 0.2816 | Helped |
| Add bias-reduced stylization (BRS) (ablation) | Riffusion-based music style transfer · Music style transfer | content preservation: 0.4415 -> 0.4645 | Helped |
| Add bias-reduced stylization (BRS) (ablation) | Riffusion-based music style transfer · Music style transfer | style fit: 0.2602 -> 0.2816 | Helped |
Surprising findings
- SS VQ-VAE reaches a style fit (0.2809) similar to the proposed method despite much worse content preservation.
Limitations
- Struggles when content has multiple components, leading to partial content loss.
- Style audio with white-noise-like sounds (rain, wind) is hard to transfer musically.
- Evaluated only on a small self-collected dataset of 253 clips.
Experimental setup
Datasets: Pixabay clips (253 five-second clips collected by authors)
Benchmarks: 282 randomly selected content-style pairs (authors' collected dataset)
Metrics: CLAP cosine similarity content preservation CLAP cosine similarity style fit subjective 1-5 ratings for content preservation, style fit,
Baselines: R+TI SS VQ-VAE MUSICGEN
Builds on: Textual Inversion Riffusion DDIM Latent Diffusion Models
Closest papers by meaning
- MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models (CVPR 2024 · similarity 0.88)
- Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversion (ICML 2024 · similarity 0.86)
- DITTO: Diffusion Inference-Time T-Optimization for Music Generation (ICML 2024 · similarity 0.85)
- MIDI-GPT: A Controllable Generative Model for Computer-Assisted Multitrack Music Composition (AAAI 2025 · similarity 0.85)
- Deep Learning for Style Transfer and Experimentation with Audio Effects and Music Creation (AAAI 2024 · similarity 0.85)
- Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.