Research map / Image generation / Text-to-image diffusion
JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal Parameters Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Customizes a pretrained text-to-music diffusion model (JEN-1) to a new instrument or genre from two minutes of reference audio by tuning only pivotal self-attention value parameters plus identifier tokens, with multiple tokens per concept for multi-concept generation.
Problem
Text-to-music models fail to reproduce user-specific musical concepts (rare instruments or genres) that text prompts cannot precisely describe; naive fine-tuning overfits to a short reference clip and multiple concepts suffer from concept conflict.
Approach
Learn a binary-like trainable mask over self-attention value projections, select the top P% most-varying parameters as pivotal and fine-tune only those (plus cross-attention K/V and identifier tokens), and use multiple identifier tokens per concept with merged masks for multi-concept training.
Key results
- Pivotal parameters tuning at 5% ratio gives Text Alignment 26.01 and Audio Alignment 39.91, versus 29.39/37.07 for the 1% default in the single-concept setting table.
- Human preference ratio of 72.3% for JEN-1 DreamStyler-Single versus 6.4%, 9.8%, and 11.5% for the three baselines.
- Using 3 identifier tokens per concept reduces Delta Audio Alignment from 24.38 (1 token) to 8.05 in two-concept generation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the identifier token only rather than tuning the DreamStyler parameter set | JEN-1 DreamStyler · Customized music generation, single-concept | Audio Alignment: 37.07 -> 27.41 | Hurt |
| fine-tune all U-Net parameters rather than use Pivotal Parameters Tuning | JEN-1 DreamStyler · Customized music generation, single-concept | Text Alignment: 29.39 -> 15.89 | Hurt |
| train only cross-attention KV and identifier tokens rather than use the DreamStyler parameter set | JEN-1 DreamStyler · Customized music generation, single-concept | Audio Alignment: 37.07 -> 23.30 | Hurt |
| increase the training parameter ratio from 1% to 5% (ablation) | Pivotal Parameters Tuning · Customized music generation, single-concept | Audio Alignment: 37.06 -> 39.91 | Helped |
| increase the training parameter ratio from 1% to 5% (ablation) | Pivotal Parameters Tuning · Customized music generation, single-concept | Text Alignment: 29.39 -> 26.01 | Hurt |
| increase the training parameter ratio from 5% to 100% (ablation) | Pivotal Parameters Tuning · Customized music generation, single-concept | Audio Alignment: 39.91 -> 46.68 | Helped |
| increase the training parameter ratio from 5% to 100% (ablation) | Pivotal Parameters Tuning · Customized music generation, single-concept | Text Alignment: 26.01 -> 18.67 | Hurt |
| select 5% of parameters randomly instead of using Pivotal Parameters selection (ablation) | JEN-1 DreamStyler · Customized music generation, single-concept | Audio Alignment: 39.91 -> 35.64 | Hurt |
| use 1 identifier token per concept instead of 3 (ablation) | JEN-1 DreamStyler · Customized music generation, multiple-concept | ΔAudio Alignment: 8.05 -> 24.38 | Hurt |
| use 5 identifier tokens per concept instead of 3 (ablation) | JEN-1 DreamStyler · Customized music generation, multiple-concept | ΔAudio Alignment: 8.05 -> 12.20 | Hurt |
| increase identifier tokens from 3 to 5 per concept (ablation) | JEN-1 DreamStyler · Customized music generation, single-concept | Audio Alignment: 37.33 -> 39.91 | Helped |
| increase identifier tokens from 3 to 5 per concept (ablation) | JEN-1 DreamStyler · Customized music generation, multiple-concept | Text Alignment: 22.25 -> 17.63 | Hurt |
Surprising findings
- Training all U-Net parameters yields the highest audio alignment (61.65) but the lowest text alignment (15.89), indicating overfitting that breaks prompt control.
- Cosine similarity between learned single identifier tokens for two concepts exceeds 99% after the text encoder, motivating multiple tokens per concept.
Limitations
- Evaluated on a small benchmark of 20 self-collected concepts and 20 prompts.
- Multi-concept integration shows a trade-off: text alignment drops when two concepts are combined (21.99-22.25 vs 26.17 single).
Experimental setup
Datasets: MusicCaps (prompts) self-collected reference clips for 20 concepts
Benchmarks: customized music generation benchmark (20 concepts, 20 MusicCaps prompts)
Metrics: Audio Alignment Score (CLAP) Text Alignment Score (CLAP) Delta Audio Alignment Preference Ratio (human)
Baselines: Train Identifier Token Only (Textual Inversion) Train All Parameters in U-Net Train Cross-Attn KV & Identifier Token (Custom Diffusion)
Builds on: JEN-1 Textual Inversion DreamBooth Custom Diffusion
Closest papers by meaning
- JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation (AAAI 2025 · similarity 0.88)
- LeVo: High-Quality Song Generation with Multi-Preference Alignment (NeurIPS 2025 · similarity 0.87)
- MusicRL: Aligning Music Generation to Human Preferences (ICML 2024 · similarity 0.87)
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.87)
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction (ICML 2026 · similarity 0.87)
- MuPT: A Generative Symbolic Music Pretrained Transformer (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.