Research map / Reinforcement learning / RL from feedback
Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models
David McAllister, Miika Aittala, Tero Karras, Janne Hellsten, Angjoo Kanazawa, Timo Aila, Samuli Laine
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| switch Flow-GRPO to the EDM-style stochastic sampler (ablation) | Flow-GRPO · Combined reward ablation | reward convergence | Improved |
| replace Flow-GRPO policy gradients with finite difference flow optimization (ablation) | Flow-GRPO with new sampler · Combined reward ablation | reward convergence | Improved |
| start all trajectories in a group from the same initial noise (ablation) | FDFO · Combined reward ablation | stability and reward | Improved |
| use more unique prompts per epoch since only two rollouts per prompt are needed (ablation) | FDFO · Combined reward ablation | reward convergence | Improved |
| deterministic sampling for one trajectory of the pair instead of stochastic for both (ablation) | FDFO · Combined reward ablation | reward convergence | No clear effect |
| use PPO clipping instead of SPO (ablation) | FDFO · Combined reward ablation | reward convergence | No clear effect |
| add normalization of the difference vector (ablation) | FDFO · Combined reward ablation | training stability | Improved |
| use true reward gradient via backprop through reward model instead of finite difference (ablation) | FDFO · Combined reward ablation | reward convergence | No clear effect |
| backpropagate reward gradient through the sampling trajectory for per-step updates (ablation) | FDFO · Combined reward ablation | reward convergence | Hurt |
| use FDFO instead of Flow-GRPO (baseline 40 steps) | SD3.5-M LoRA · Combined reward wall-clock | GPU hours to target reward: 19x faster | Improved |
| use FDFO instead of Flow-GRPO (fast 10 steps) | SD3.5-M LoRA · Combined reward wall-clock | GPU hours to target reward: 5x faster | Improved |
| use combined reward instead of PickScore alone | FDFO and Flow-GRPO · OneIG-Bench alignment | prompt alignment | Improved |
Closest papers by meaning
- Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.91)
- Flow-GRPO: Training Flow Matching Models via Online RL (NeurIPS 2025 · similarity 0.90)
- David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.90)
- Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL (NeurIPS 2026 · similarity 0.88)
- GARDO: Reinforcing Diffusion Models without Reward Hacking (NeurIPS 2026 · similarity 0.88)
- Learning to Sample Effective and Diverse Prompts for Text-to-Image Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
