Research map / Image generation / Fast diffusion sampling
Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ∇-GFlowNet (residual ∇-DB), a gradient-informed GFlowNet fine-tuning method that aligns Stable Diffusion v1.5 to reward models while preserving diversity and pretrained prior.
Problem
Reward finetuning of pretrained diffusion models often loses sample diversity, forgets the pretrained prior, or converges slowly.
Approach
Derives gradient-informed detailed balance conditions (taking derivatives of DB in log space w.r.t. states) and a residual variant relative to the pretrained model, so reward gradients are used while sampling is proportional to R(x)^beta times the pretrained prior; adds forward-looking parameterization of the residual flow score.
Key results
- On Aesthetic Score, nabla-GFlowNet (w_B=1) reaches reward 7.90 with DreamSim diversity 29.67 and FID 317, versus DDPO 6.68 / 32.96 / 312 and DRaFT-LV 10.21 / 6.39 / 1854.
- On ImageReward, nabla-GFlowNet (w_B=1) achieves reward 0.223 with diversity 39.85 and FID 501, outperforming DAG-DB diversity 26.78 and residual DB FID 772.
- Gradient-based baselines ReFL and DRaFT collapse in diversity (DreamSim ~4-8 on Aesthetic Score) while the proposed method keeps diversity near the base model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ∇-GFlowNet instead of the base model | StableDiffusion-v1.5 · Aesthetic Score | reward: 5.83 $\scriptscriptstyle\pm$ { 0.01} -> 7.90 $\scriptscriptstyle\pm$ { 0.09} | Helped |
| Use ∇-GFlowNet instead of the base model | StableDiffusion-v1.5 · Aesthetic Score | DreamSim diversity: 35.91 $\scriptscriptstyle\pm$ { 0.00} -> 29.67 $\scriptscriptstyle\pm$ { 0.51} | Hurt |
| Use ∇-GFlowNet instead of the base model | StableDiffusion-v1.5 · HPSv2 | reward: 2.38 $\scriptscriptstyle\pm$ { 0.13} -> 3.53 $\scriptscriptstyle\pm$ { 0.04} | Helped |
| Use ∇-GFlowNet instead of the base model | StableDiffusion-v1.5 · ImageReward | reward: -0.38 $\scriptscriptstyle\pm$ { 0.12} -> 2.23 $\scriptscriptstyle\pm$ { 0.34} | Helped |
| Set w_B=1 rather than w_B=0 (ablation) | ∇-GFlowNet · Aesthetic Score | DreamSim diversity: 29.21 $\scriptscriptstyle\pm$ { 0.18} -> \textbf{29.67} $\scriptscriptstyle\pm$ { 0.51} | Helped |
| Set w_B=1 rather than w_B=0 (ablation) | ∇-GFlowNet · HPSv2 | FID: 973 $\scriptscriptstyle\pm$ { 5} -> 1000 $\scriptscriptstyle\pm$ { 39} | Hurt |
| Set w_B=1 rather than w_B=0 (ablation) | ∇-GFlowNet · ImageReward | reward: 5.33 $\scriptscriptstyle\pm$ { 0.62} -> 2.23 $\scriptscriptstyle\pm$ { 0.34} | Hurt |
| Use higher reward temperature β (ablation) | residual ∇-DB · Aesthetic Score | reward convergence speed | Helped |
| Use higher reward temperature β (ablation) | residual ∇-DB · Aesthetic Score | diversity and prior preservation | Hurt |
| Subsample transitions in each trajectory (ablation) | residual ∇-DB · Aesthetic Score | performance | No clear effect |
| Remove attenuation scaling on predicted reward (ablation) | residual ∇-DB · Aesthetic Score | convergence speed | Helped |
| Remove attenuation scaling on predicted reward (ablation) | residual ∇-DB · Aesthetic Score | diversity and prior preservation | Hurt |
Surprising findings
- Gradient-informed baselines ReFL and DRaFT achieve higher reward but suffer severe mode collapse and prior forgetting.
- Second-order gradients were found to hurt the reward/diversity/prior trade-off.
Limitations
- Over-optimization of reward still requires careful tuning of reward temperature beta and output regularization strength lambda.
- Flow-matching extension is only described in theory, not experimentally validated.
Experimental setup
Datasets: DDPO animal prompts HPDv2
Benchmarks: Aesthetic Score HPSv2 ImageReward
Metrics: reward DreamSim diversity FID
Baselines: DDPO ReFL DRaFT-1 DRaFT-LV DAG-DB residual DB
Builds on: GFlowNet detailed balance DDPO DAG-DB forward-looking GFlowNet DRaFT ReFL
Closest papers by meaning
- Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.90)
- Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewards (NeurIPS 2025 · similarity 0.89)
- On Divergence Measures for Training GFlowNets (NeurIPS 2024 · similarity 0.89)
- Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.89)
- Training Diffusion Models Towards Diverse Image Generation with Reinforcement Learning (CVPR 2024 · similarity 0.88)
- Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.