Research map / Image generation / Fast diffusion sampling
Diversity-Rewarded CFG Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills classifier-free guidance into a music LLM via on-policy KL distillation, adds an RL diversity reward based on embedding cosine dissimilarity, and merges quality- and diversity-focused weights by linear interpolation.
Problem
Classifier-free guidance (CFG) improves output quality but doubles inference cost and reduces diversity of generated content, particularly for creative generation such as text-to-music.
Approach
Finetune an autoregressive model to match its own CFG-augmented logits on on-policy samples (CFG distillation), jointly maximize a diversity reward computed as negative cosine similarity between embeddings of multiple samples via a REINFORCE-style estimator, and interpolate the weights of quality- and diversity-focused finetunes to steer the trade-off at deployment time.
Key results
- Merged model LERP(0, 15) is more diverse than the CFG-augmented base model (gamma=3) in 57% of human diversity comparisons while rated at 51% for quality.
- CFG-distilled model (beta=0) matches CFG-augmented base on human quality win rate (0.50), both beating the base model (0.66 and 0.68).
- The beta=15 diversity-rewarded model achieves 0.73 and 0.79 win rates on diversity against CFG and CFG-distilled beta=0 models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill CFG (γ=3) into model weights without inference-time CFG | MusicRL-R · Text-to-music generation | quality | Helped |
| Distill CFG without a diversity reward (β=0) (ablation) | MusicRL-R · Text-to-music generation | diversity | Hurt |
| Add a diversity reward with β=10 or β=15 (ablation) | Diversity-rewarded CFG distillation · Text-to-music generation | diversity | Helped |
| Increase the diversity-reward coefficient β (ablation) | Diversity-rewarded CFG distillation · Text-to-music generation | quality | Hurt |
| Interpolate weights of the quality-focused β=0 and diversity-focused β=15 models | Diversity-rewarded CFG distillation · Text-to-music quality-diversity trade-off | quality at a given diversity | Helped |
| Interpolate weights of the β=0 and β=15 models with λ=0.5 | Diversity-rewarded CFG distillation · Text-to-music human evaluation vs CFG-augmented base | diversity: 57% | Helped |
| Interpolate weights of the β=0 and β=15 models with λ=0.5 | Diversity-rewarded CFG distillation · Text-to-music human evaluation vs CFG-augmented base | quality: 51% | No clear effect |
| Increase CFG guidance from γ=1 to γ=7 | MusicRL-R · Text-to-music generation | diversity: 0.5 -> 0.3 | Hurt |
| Use negative prompting with “Bad audio quality.” instead of unconditional prompting | CFG on MusicRL-R · Text-to-music quality-diversity trade-off | quality-diversity trade-off | Helped |
| Apply CFG or temperature sampling to the already CFG-distilled model | CFG-distilled MusicRL-R · Text-to-music generation | quality and diversity | No clear effect |
| Use CFG-distilled β=15 rather than the CFG-distilled β=0 or CFG baseline (ablation) | MusicRL-R · Text-to-music human evaluation | diversity: 0.73; 0.79 | Helped |
Surprising findings
- The beta=15 model scored on par with the base model in human diversity evaluation (0.50 win rate), despite higher automatic diversity scores, attributed to reward hacking.
- Diversity-maximizing models sometimes generate excessive background drums, likely from a bias of the embedding model.
Limitations
- Diversity measure (embedding cosine dissimilarity) can be hacked, e.g. excessive background drums.
- Evaluated only on text-to-music with a single base model (MusicRL-R, not publicly available).
- Automatic diversity metric disagrees with human diversity judgments for beta=15 model.
Experimental setup
Datasets: MusicRL prompt dataset MusicCaps contrastive music embedding training data (same as MusicLM)
Benchmarks: MuLan score (text adherence) User Preference score diversity via embedding cosine dissimilarity human side-by-side evaluation
Metrics: quality score (5*MuLan + User Preference) diversity (1 - cosine similarity) win rate KL divergence
Baselines: CFG (guidance factor sweep) temperature sampling base MusicRL-R model LERP(0, base) uniform LERP(0,5,10,15)
Builds on: classifier-free guidance on-policy distillation MusicRL linear mode connectivity REINFORCE
Closest papers by meaning
- MusicRL: Aligning Music Generation to Human Preferences (ICML 2024 · similarity 0.85)
- Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation (ICML 2025 · similarity 0.85)
- JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal Parameters Tuning (AAAI 2025 · similarity 0.85)
- Presto! Distilling Steps and Layers for Accelerating Music Generation (ICLR 2025 · similarity 0.85)
- DITTO: Diffusion Inference-Time T-Optimization for Music Generation (ICML 2024 · similarity 0.84)
- Multi-Source Diffusion Models for Simultaneous Music Generation and Separation (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.