Research map / Reinforcement learning / RL from feedback
Reward modeling versus direct optimization
Compare reward-model RLHF with DPO-style direct policy optimization and refine preference objectives.
95 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in RL from feedback, keyword cluster “human · rlhf · preferences”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
20 papers in 2023–24 and 71 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 22.0% to 11.5%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Deep Reinforcement Learning from Human Preferences (NeurIPS 2017 · 500 citations)
- 2019 · Text-Based Interactive Recommendation via Constraint-Augmented Reinforcement Learning (NeurIPS 2019 · 29 citations)
- 2020 · What Makes A Good Story? Designing Composite Rewards for Visual Storytelling (AAAI 2020 · 56 citations)
- 2022 · SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning (ICLR 2022 · 14 citations)
- 2023 · Aligning Language Models with Preferences through $f$-divergence Minimization (ICML 2023 · 5 citations)
- 2024 · Towards Automated RISC-V Microarchitecture Design with Reinforcement Learning (AAAI 2024 · 13 citations)
- 2025 · Radiology Report Generation via Multi-objective Preference Optimization (AAAI 2025 · 9 citations)
- 2026 · Human-in-the-Loop Policy Optimization for Preference-Based Multi-Objective Reinforcement Learning (ICML 2026 · 2 citations)
Earliest papers here
- Deep Reinforcement Learning from Human Preferences (NeurIPS 2017 · 500 citations)
- Text-Based Interactive Recommendation via Constraint-Augmented Reinforcement Learning (NeurIPS 2019 · 29 citations)
- What Makes A Good Story? Designing Composite Rewards for Visual Storytelling (AAAI 2020 · 56 citations)
What has been tried
54 papers here report 510 measured changes (a component added, removed or swapped, compared with a baseline): helped 63%, no clear effect 7%, hurt 23%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add proximal updates | 45 animal prompts · training stability | Helped | PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models (CVPR 2024 · 12 citations) | |
| Increase the number of rating classes beyond 3 in RbRL | Quadruped synthetic experiment · mean reward | Hurt | Rating-Based Reinforcement Learning (AAAI 2024 · 11 citations) | |
| add MOO to the Base model | IU-Xray · BLEU-1 | 0.470 -> 0.540 | Helped | Radiology Report Generation via Multi-objective Preference Optimization (AAAI 2025 · 9 citations) |
| use tournament ranking instead of first-round ranking | Reddit TL;DR · Proxy Reward (%) | 92.37 -> 93.35 | Helped | Statistical Rejection Sampling Improves Preference Optimization (ICLR 2024 · 7 citations) |
| Add unified policy distillation in place of independent per-anchor score regression | NAVSIM Navtest · PDMS | 87.3 -> 88.8 | Helped | DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving (NeurIPS 2025 · 3 citations) |
| Freeze the reward model partway through training | Continuous control preference learning · Performance | Hurt | A Minimaximalist Approach to Reinforcement Learning from Human Feedback (ICML 2024 · 3 citations) | |
| add forward KL regularization rather than reverse KL regularization | MetaWorld 20k Sparse, Door Open and Drawer Open; Button Press · improvement factor | Door Open: 40.0±{3.5} vs 21.7±{8.8}; Drawer Open: 47.2±{3.7} vs 6.9±{3.4}; Button Press: 58.6±{21.6} | Mixed | Forward KL Regularized Preference Optimization for Aligning Diffusion Policies (AAAI 2025 · 2 citations) |
| Increase β during QED alignment | QED · QED concentration near 1 | Helped | Aligning Transformers with Continuous Feedback via Energy Rank Alignment (NeurIPS 2025 · 2 citations) | |
| Increase the regularisation parameter τ | Summarisation · performance against RL | Mixed | Human Alignment of Large Language Models through Online Preference Optimisation (ICML 2024 · 2 citations) | |
| Replace DPO safety realignment with KTO after DPO helpfulness alignment | Helpfulness and harmlessness evaluations · win rate vs SFT | Hurt | Stepwise Alignment for Constrained Language Model Policy Optimization (NeurIPS 2024 · 2 citations) | |
| Use value-based alignment instead of preference-based alignment | 9 D4RL Locomotion tasks · normalized score | Helped | Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control (NeurIPS 2024 · 2 citations) | |
| using a larger monotonicity-check ε (ε_c=0.02) with ε_s=0.01 instead of shared ε=0.01 | Anthropic-HH win rate vs chosen · win rate (%) | 74.2 vs 79.2 default | Hurt | KL Penalty Control via Perturbation for Direct Preference Optimization (NeurIPS 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 40 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- UltraFeedback · 9 of 40 papers
- Anthropic-HH · 3 of 40 papers
- HelpSteer2 · 3 of 40 papers
- TSPLIB · 2 of 40 papers
- GSM8K · 2 of 40 papers
- MultiPref · 2 of 40 papers
- Reddit TL;DR · 2 of 40 papers
- HH-RLHF · 2 of 40 papers
Baselines
- DPO · 17 of 40 papers
- IPO · 8 of 40 papers
- KTO · 7 of 40 papers
- SimPO · 7 of 40 papers
- SFT · 6 of 40 papers
- R-DPO · 4 of 40 papers
- TDPO · 4 of 40 papers
- CPO · 3 of 40 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 91 papers; from OpenAlex and the papers' affiliations).
- Google · 10 papers
- Tsinghua University · 5 papers
- Hong Kong University of Science and Technology · 3 papers
- Korea Advanced Institute of Science and Technology · 3 papers
- Carnegie Mellon University · 3 papers
- Meta · 3 papers
- DEVCOM Army Research Laboratory · 2 papers
- The Chinese University of Hong Kong · 2 papers
- Institute of Artificial Intelligence · 2 papers
- Accenture · 2 papers
Use it
Download BibTeX of 27 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Theory of RLHF (110 papers)
- Preference-based RL reward learning (26 papers)
Elsewhere in the topic
- RL for LLM reasoning (187 papers)
- Scaling RL with verifiable rewards (119 papers)
- GRPO stability and variants (106 papers)
- RL fine-tuning of generative models (94 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).