Research map / Image generation / Fast diffusion sampling
Value Gradient Guidance for Flow Matching Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VGG-Flow, an optimal-control (HJB) based gradient-matching method that finetunes pretrained flow matching models toward reward objectives while preserving the base prior, demonstrated on Stable Diffusion 3.
Problem
Aligning pretrained flow matching models with human-preference reward models efficiently while preserving the base model's prior and sample diversity; existing reward finetuning methods either overfit or are expensive (adjoint matching), and probability-flow-based gradient methods from diffusion do
Approach
Relax the alignment objective to reward plus an L2 control cost on the residual velocity field relative to the base model; the HJB optimality condition gives a gradient-matching law (residual velocity = -value gradient / lambda), with the value gradient learned via a consistency PDE and parameterized with a forward-looking one-step Euler reward gradient plus a learned correction.
Key results
- On Aesthetic Score, VGG-Flow reaches reward 8.24 with FID 375 versus 465 for adjoint matching and 1338 for ReFL at similar or lower prior preservation (base SD3 FID 212 reference).
- On HPSv2, VGG-Flow attains reward 3.86 and DreamSim diversity 18.40, versus 3.87/14.08 for ReFL and 3.76/15.05 for DRaFT.
- On PickScore, VGG-Flow reaches reward 23.21 with diversity 20.93, outperforming DRaFT (23.00, 19.03) and adjoint matching (22.78, 19.70) at 400 update steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with VGG-Flow instead of the base model | Stable Diffusion 3 · Aesthetic Score | reward: 5.99 ± { 0.01} -> 8.24 ± { 0.07} | Helped |
| Fine-tune with VGG-Flow instead of the base model | Stable Diffusion 3 · HPSv2 | reward: 2.80 ± { 0.05} -> 3.86 ± { 0.03} | Helped |
| Fine-tune with VGG-Flow instead of the base model | Stable Diffusion 3 · PickScore | reward: 21.81 ± { 0.02} -> 23.21 ± { 0.05} | Helped |
| Use a higher reward temperature β (ablation) | VGG-Flow · Aesthetic Score | reward convergence speed | Helped |
| Use a higher reward temperature β (ablation) | VGG-Flow · Aesthetic Score | sample diversity | Hurt |
| Use a higher reward temperature β (ablation) | VGG-Flow · Aesthetic Score | FID vs. reward trade-off | Hurt |
| Use η_t = t instead of η_t = t² (ablation) | VGG-Flow · Aesthetic Score | convergence speed | Helped |
| Use η_t = t instead of η_t = t² (ablation) | VGG-Flow · Aesthetic Score | metric trade-offs | No clear effect |
| Reduce transition subsampling rate from 50% to 25% (ablation) | VGG-Flow · Aesthetic Score | performance | No clear effect |
Surprising findings
- ReFL and DRaFT on Aesthetic Score reach reward values near 9, which the authors say typically indicates complete forgetting of the base prior.
- Adjoint matching sometimes retains higher diversity than VGG-Flow on Aesthetic Score (22.34 vs 22.12), though with worse reward.
Limitations
- Relaxed objective means the finetuned distribution approximates the ideal KL-regularized target only for small lambda.
- Finite-difference approximations with disabled second-order gradients introduce bias.
- Prone to mode collapse and exploration-exploitation tradeoffs under the limited-compute hyperparameter settings; architecture design not explored.
Experimental setup
Datasets: DDPO animal prompts HPDv2 photo+painting prompts Pic-a-Pick prompt set
Benchmarks: Aesthetic Score HPSv2 PickScore
Metrics: reward DreamSim diversity CLIP diversity per-prompt FID
Baselines: ReFL DRaFT Adjoint Matching VGG-Flow-PMP
Builds on: flow matching adjoint matching DRaFT ReFL DreamFusion
Closest papers by meaning
- On the Guidance of Flow Matching (ICML 2025 · similarity 0.92)
- Optimal Flow Matching: Learning Straight Trajectories in Just One Step (NeurIPS 2024 · similarity 0.90)
- Flow-GRPO: Training Flow Matching Models via Online RL (NeurIPS 2025 · similarity 0.90)
- Improving Flow Matching by Aligning Flow Divergence (ICML 2025 · similarity 0.89)
- Diff2Flow: Training Flow Matching Models via Diffusion Model Alignment (CVPR 2025 · similarity 0.89)
- Structured Coupling for Flow Matching (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.