Research map / Reinforcement learning / RL from feedback
RL fine-tuning of generative models
Fine-tune diffusion and multimodal generative models with policy-gradient and reward objectives.
94 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in RL from feedback, keyword cluster “reasoning · grpo · llm”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
7 papers in 2023–24 and 85 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 7.7% to 13.7%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (ICML 2017 · 73 citations)
- 2022 · On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (NeurIPS 2022)
- 2023 · DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models (NeurIPS 2023 · 39 citations)
- 2024 · Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment (ICLR 2024 · 7 citations)
- 2025 · Flow-GRPO: Training Flow Matching Models via Online RL (NeurIPS 2025 · 36 citations)
- 2026 · GUI-G²: Gaussian Reward Modeling for GUI Grounding (AAAI 2026 · 3 citations)
Earliest papers here
- Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (ICML 2017 · 73 citations)
- On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (NeurIPS 2022)
- DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models (NeurIPS 2023 · 39 citations)
What has been tried
22 papers here report 239 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 5%, hurt 23%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add KL regularization during compositional-generation training | DrawBench, compositional generation training · DeQA | 2.77 -> 4.01 | Helped | Flow-GRPO: Training Flow Matching Models via Online RL (NeurIPS 2025 · 36 citations) |
| add off-policy RL fine-tuning (skating-aware step rewards) | DD100 · FID_k | 106.72 -> 25.30 (81.42 lower) | Helped | Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment (ICLR 2024 · 7 citations) |
| Use dense point rewards instead of sparse rewards with filtered data | ScreenSpot-Pro · average accuracy | 35.8 -> 40.0 | Helped | SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · 4 citations) |
| Use LaSRO rather than direct Image Reward gradients | MJHQ-30K · Image Reward | Helped | Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward (CVPR 2025 · 2 citations) | |
| Use one REBEL iteration instead of DPO on online best-of-5/worst-of-5 data | UltraFeedback · LC win rate | 44.9 -> 48.3 | Helped | REBEL: Reinforcement Learning via Regressing Relative Rewards (NeurIPS 2024 · 2 citations) |
| Use clipping range 2 × β₀ instead of recommended 1 × β₀ | MMAU · Total accuracy (%) | 76.0 -> 75.7 | Hurt | Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models (NeurIPS 2025 · 1 citation) |
| add W2 regularization | SD3 text-image alignment, “a cat in the sky” · CLIP Score | 33.63 -> 35.46 | Helped | Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization (ICLR 2025 · 1 citation) |
| remove KL regularization | Regulatory DNA design (HepG2) · Pred-Activity (median) | 5.61(0.07) -> 6.44(0.04) | Helped | Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design (ICLR 2025 · 1 citation) |
| Use SEIKO Bootstrap instead of Greedy, adding KL regularization and the bootstrap uncertainty bonus | GFP · Value | 3.62 ± 0.00 -> \mathbf{3.87± 0.01} | Helped | Feedback Efficient Online Fine-Tuning of Diffusion Models (ICML 2024 · 1 citation) |
| Replace RM(x,c) with RM(x̂θ,c) in the reward mean predictor | Value network pretraining · MSE | 2.63 -> 0.67 | Helped | Score as Action: Fine Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning (ICML 2025 · 0 citations) |
| pre-train on multi-task MT-10 data rather than basketball only | Unseen tasks: drawer-open, plate-slide-side, handle-pull-side · success rate | 34.7 (± 0.06) vs 0.0 (± 0.0); 55.3 (± 0.33) vs 0.0 (± 0.0); 71.3 (± 0.13) vs 0.0 (± 0.0) | Helped | Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Planner (ICML 2025 · 0 citations) |
| Use selective merging weights rather than uniform fixed merging weights | DNA sequence generation (RLOO) · Avg Reward | No clear effect | Iterative Foundation Model Fine-Tuning on Multiple Rewards (NeurIPS 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 36 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- DrawBench · 3 of 36 papers
- T2I-CompBench · 2 of 36 papers
- CIFAR-10 · 2 of 36 papers
- HH-RLHF · 2 of 36 papers
- Pick-a-Pic · 2 of 36 papers
Baselines
- DDPO · 5 of 36 papers
- GRPO · 4 of 36 papers
- Flow-GRPO · 4 of 36 papers
- FLUX.1-Dev · 3 of 36 papers
- Base model · 3 of 36 papers
- SDXL-Lightning · 2 of 36 papers
- GPT-4o · 2 of 36 papers
- DPO · 2 of 36 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 92 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 5 papers
- Google DeepMind · 4 papers
- University of California, Berkeley · 4 papers
- Hong Kong University of Science and Technology · 4 papers
- Carnegie Mellon University · 4 papers
- The Chinese University of Hong Kong · 4 papers
- Tencent · 4 papers
- ByteDance · 3 papers
- Google · 3 papers
- Princeton University · 3 papers
Use it
Download BibTeX of 24 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- RL for LLM reasoning (187 papers)
- Scaling RL with verifiable rewards (119 papers)
- GRPO stability and variants (106 papers)
Elsewhere in the topic
- Theory of RLHF (110 papers)
- Reward modeling versus direct optimization (95 papers)
- Preference-based RL reward learning (26 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).