Research map / Reinforcement learning / RL from feedback
Flow-GRPO: Training Flow Matching Models via Online RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Flow-GRPO brings online GRPO-style policy gradient RL to flow matching text-to-image models by converting the deterministic ODE sampler into an SDE with matching marginals and reducing training denoising steps, lifting SD3.5-M GenEval from 0.63 to 0.95.
Problem
Flow matching text-to-image models use deterministic ODE sampling, which precludes the stochastic exploration online RL needs, and their many denoising steps make RL sample collection expensive.
Approach
Convert the deterministic flow ODE into a reverse-time SDE that preserves marginal distributions so each denoising step is a Gaussian policy amenable to GRPO, and cut training denoising steps (T=10) while keeping the full 40-step schedule at inference.
Key results
- SD3.5-M+Flow-GRPO raises GenEval overall accuracy from 0.63 to 0.95, above GPT-4o (0.84).
- Visual text rendering OCR accuracy rises from 0.59 to 0.92 with KL regularization.
- Reducing training timesteps from 40 to 10 gives over 4x speedup without impacting final reward on the ablated tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SD3.5-M online with Flow-GRPO | SD3.5-M · GenEval test | Overall accuracy: 0.63 -> 0.95 | Helped |
| Fine-tune SD3.5-M online with Flow-GRPO | SD3.5-M · Visual Text Rendering test | OCR accuracy: 0.59 -> 0.92 | Helped |
| Fine-tune SD3.5-M online with Flow-GRPO | SD3.5-M · Human Preference Alignment test | PickScore: 21.72 -> 23.31 | Helped |
| Add KL regularization during compositional-generation training (ablation) | Flow-GRPO · DrawBench, compositional generation training | DeQA: 2.77 -> 4.01 | Helped |
| Add KL regularization during compositional-generation training (ablation) | Flow-GRPO · DrawBench, compositional generation training | ImageReward: 0.44 -> 1.03 | Helped |
| Add KL regularization during text-rendering training (ablation) | Flow-GRPO · Visual Text Rendering test | OCR accuracy: 0.93 -> 0.92 | Hurt |
| Add KL regularization during human-preference training (ablation) | Flow-GRPO · Human Preference Alignment test | PickScore: 23.41 -> 23.31 | No clear effect |
| Reduce training data-collection denoising steps from 40 to 10 (ablation) | Flow-GRPO · GenEval, Visual Text Rendering, and Human Preference Alignment | training speed: over a $4\times$ speedup | Helped |
| Reduce training data-collection denoising steps from 40 to 10 (ablation) | Flow-GRPO · GenEval, Visual Text Rendering, and Human Preference Alignment | final reward | No clear effect |
| Reduce training data-collection denoising steps from 40 to 5 (ablation) | Flow-GRPO · GenEval, Visual Text Rendering, and Human Preference Alignment | training speed | Mixed |
| Increase SDE noise level from a=0.1 toward a=0.7 (ablation) | Flow-GRPO · reward optimization | reward improvement rate | Helped |
| Increase SDE noise level beyond a=0.7 toward a=1.0 (ablation) | Flow-GRPO · reward optimization | reward improvement rate | No clear effect |
Surprising findings
- Low-quality 10-step SDE training samples with visible artifacts still accelerate optimization, because GRPO relies on relative preferences.
- Removing KL on the PickScore task did not reduce image quality metrics but caused visual diversity to collapse.
Limitations
- Focused on text-to-image; video extension left to future work.
- KL regularization requires longer training and reward hacking still occurs for certain prompts.
- Reward design relies on rule-based or model-based rewards; more advanced reward models needed for video.
Experimental setup
Datasets: GenEval training prompts (official scripts) GPT-4o-generated text rendering prompts (20K train / 1K test) PickScore prompts DrawBench T2I-CompBench++
Benchmarks: GenEval OCR accuracy (visual text rendering) PickScore DrawBench T2I-CompBench++
Metrics: GenEval overall accuracy OCR accuracy Aesthetic score DeQA ImageReward PickScore UnifiedReward CLIP score diversity score
Baselines: SD3.5-M GPT-4o FLUX.1 Dev SD3.5-L SANA-1.5 4.8B Janus-Pro-7B Flow-DPO Flow-RWR DDPO ReFL ORW SFT
Builds on: GRPO DDPO Rectified Flow Flow-DPO PPO
Closest papers by meaning
- Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models (NeurIPS 2026 · similarity 0.91)
- PRECISE: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models (NeurIPS 2026 · similarity 0.91)
- Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation (NeurIPS 2025 · similarity 0.90)
- Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization (ICLR 2025 · similarity 0.90)
- Value Gradient Guidance for Flow Matching Alignment (NeurIPS 2025 · similarity 0.90)
- Flow-Based Policy for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.