Research map / Image generation / Text-to-image diffusion
DiffExp: Efficient Exploration in Reward Fine-tuning for Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffExp, an exploration strategy for reward fine-tuning of text-to-image diffusion models that dynamically schedules CFG scale and randomly up-weights prompt tokens, improving sample efficiency of DDPO and AlignProp.
Problem
Reward fine-tuning of text-to-image diffusion models with online sample generation converges slowly because generated samples often lack strong reward signals, so exploration is insufficient.
Approach
Combine a dynamic CFG schedule (very low guidance scale for early denoising steps, ordinary scale afterwards) with random token-embedding weighting of the prompt during online sampling, to increase sample diversity while preserving quality and surfacing high-reward samples.
Key results
- DiffExp reaches the same reward as DDPO and AlignProp with roughly 20% fewer samples, per the paper's claim.
- DiffExp yields higher unseen ImageReward scores than DDPO and AlignProp fine-tuned with Aesthetic or PickScore.
- AlignProp with DiffExp achieves higher maximum Aesthetic reward and better sample efficiency when fine-tuning SDXL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DiffExp exploration to DDPO | DDPO fine-tuning of Stable Diffusion v1.5 · Aesthetic and PickScore reward fine-tuning | reward at a given number of reward queries: approximately 20% fewer samples to achieve the same reward score | Helped |
| Add DiffExp exploration to AlignProp | AlignProp fine-tuning of Stable Diffusion v1.5 · Aesthetic and PickScore reward fine-tuning | reward at a given number of reward queries: approximately 20% fewer samples to achieve the same reward score | Helped |
| Add DiffExp exploration | DDPO and AlignProp fine-tuned with Aesthetic or PickScore · Unseen ImageReward evaluation | ImageReward | Helped |
| Add DiffExp exploration | DDPO and AlignProp · Unseen prompts | reward | Helped |
| Add DiffExp exploration | AlignProp · DrawBench prompts | reward | Helped |
| Add DiffExp exploration | AlignProp fine-tuning of SDXL · Aesthetic reward fine-tuning | maximum reward | Helped |
| Add random prompt weighting and dynamic CFG scheduling together (ablation) | AlignProp · Aesthetic and PickScore reward fine-tuning | reward | Helped |
| Remove random prompt weighting while retaining dynamic CFG scheduling (ablation) | DiffExp + AlignProp · Aesthetic and PickScore reward fine-tuning | reward | Hurt |
| Remove dynamic CFG scheduling while retaining random prompt weighting (ablation) | DiffExp + AlignProp · Aesthetic and PickScore reward fine-tuning | reward | Hurt |
| Use t_thres=900 instead of a lower tested threshold (ablation) | DiffExp + AlignProp · Aesthetic and PickScore reward fine-tuning | reward: t_thres={900, 800, 700} | Helped |
| Lower t_thres from 900 to 800 or 700 (ablation) | DiffExp + AlignProp · Aesthetic and PickScore reward fine-tuning | image quality: t_thres={900, 800, 700} | Hurt |
Surprising findings
- Higher CFG-switch threshold (t_thres=900) gave the best results, and lowering it degraded image quality.
Limitations
- Gains are reported as sample-efficiency curves on small prompt sets (45 animals, 135 animal-activity prompts, 58 DrawBench prompts).
- Random prompt weighting range and CFG schedule are hand-chosen hyperparameters (e.g., w_prompt in U(1,1.2), t_thres=900).
Experimental setup
Datasets: DrawBench Parti prompts (unseen evaluation)
Benchmarks: DrawBench Parti prompts
Metrics: reward score (Aesthetic, PickScore, HPS) ImageReward (unseen reward) sample efficiency (reward vs. number of reward queries)
Baselines: DDPO AlignProp
Builds on: DDPO AlignProp classifier-free guidance LoRA
Closest papers by meaning
- GARDO: Reinforcing Diffusion Models without Reward Hacking (NeurIPS 2026 · similarity 0.89)
- Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.89)
- Directly Fine-Tuning Diffusion Models on Differentiable Rewards (ICLR 2024 · similarity 0.89)
- Self-Play Fine-tuning of Diffusion Models for Text-to-image Generation (NeurIPS 2024 · similarity 0.87)
- Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models (NeurIPS 2026 · similarity 0.87)
- Efficient Adjoint Matching for Fine-tuning Diffusion Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.