Research map / Reinforcement learning / RL from feedback
Theory of RLHF
Analyze sample efficiency and reward-model behavior in reinforcement learning from human feedback.
110 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in RL from feedback, keyword cluster “human · rlhf · preferences”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
40 papers in 2023–24 and 61 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 44.0% to 9.8%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Interactive Learning from Policy-Dependent Human Feedback (ICML 2017 · 104 citations)
- 2020 · Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards (AAAI 2020 · 15 citations)
- 2021 · Learning Rewards From Linguistic Feedback (AAAI 2021 · 20 citations)
- 2022 · Reward Uncertainty for Exploration in Preference-based Reinforcement Learning (ICLR 2022 · 8 citations)
- 2023 · Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (ICLR 2023 · 53 citations)
- 2024 · Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model (CVPR 2024 · 49 citations)
- 2025 · Alleviating Shifted Distribution in Human Preference Alignment through Meta-Learning (AAAI 2025 · 1 citation)
- 2026 · How RLHF Amplifies Sycophancy (ICML 2026 · 2 citations)
Earliest papers here
- Interactive Learning from Policy-Dependent Human Feedback (ICML 2017 · 104 citations)
- Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards (AAAI 2020 · 15 citations)
- What Is It You Really Want of Me? Generalized Reward Learning with Biased Beliefs about Domain Dynamics (AAAI 2020 · 6 citations)
What has been tried
76 papers here report 739 measured changes (a component added, removed or swapped, compared with a baseline): helped 65%, no clear effect 13%, hurt 17%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use VPL rather than BTL on a unimodal preference dataset | UF-P-4 unimodal setting · evaluation accuracy | 77.04% -> 77.16% | No clear effect | Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · 10 citations) |
| use few-shot examples from the skills library instead of zero-shot generation | ManiSkill2 and MetaWorld manipulation tasks · downstream performance | Helped | Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · 9 citations) | |
| shift the learned optimal advantage so its maximum action value is 0 in every state | 90 small gridworld MDPs · normalized mean return | No clear effect | Learning Optimal Advantage from Preferences and Mistaking It for Reward (AAAI 2024 · 5 citations) | |
| replace DPO loss with adaptive preference scaling and quadratic regularization | Dialogue · win rate | 53.38 -> 56.00 | Helped | Adaptive Preference Scaling for Reinforcement Learning with Human Feedback (NeurIPS 2024 · 4 citations) |
| Add two-stage sampling | IWSLT development set · Memory (G) | >24.00 -> 16.35 | Helped | ESRL: Efficient Sampling-Based Reinforcement Learning for Sequence Generation (AAAI 2024 · 3 citations) |
| Use full ranking information with m = k rather than top-1 ranking with k = 1 | Quadratic function and Rosenbrock function · optimization performance | Helped | Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles (ICLR 2024 · 3 citations) | |
| add predicted keypoints as subgoals | maze2d-medium-v1 and maze2d-large-v1 · normalized score | maze2d-medium-v1: 59.5±36.3 -> 131.7±33.3; maze2d-large-v1: 97.1±25.4 -> 150.6±60.2 | Helped | Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · 3 citations) |
| Increase the number of averaged reward models | TL;DR RL · control reward | Helped | WARM: On the Benefits of Weight Averaged Reward Models (ICML 2024 · 3 citations) | |
| Increase reward-model size | BoN and PPO, 25% label noise · gold reward model performance | Helped | Reward Model Ensembles Help Mitigate Overoptimization (ICLR 2024 · 2 citations) | |
| Use generator-model attention instead of reward-model attention (ABC-D) | Positive Generation · average reward | Mixed | Dense Reward for Free in Reinforcement Learning from Human Feedback (ICML 2024 · 2 citations) | |
| Increase the dual variable λ | PKU-SafeRLHF-30K test; model- and GPT-based evaluation · safety score / safety win rate | Helped | One-Shot Safety Alignment for Large Language Models via Optimal Dualization (NeurIPS 2024 · 2 citations) | |
| Vary MetaRM meta-process learning rate η | Anthropic-Harmless, Anthropic-Helpful, Summary · win rate | No clear effect | Alleviating Shifted Distribution in Human Preference Alignment through Meta-Learning (AAAI 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 40 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- UltraFeedback · 4 of 40 papers
- TL;DR · 3 of 40 papers
- ShareGPT · 2 of 40 papers
- Reddit TL;DR · 2 of 40 papers
- PKU-SafeRLHF · 2 of 40 papers
- TL;DR summarization preference dataset · 2 of 40 papers
- AlpacaFarm · 2 of 40 papers
- Meta-World · 2 of 40 papers
Baselines
- DPO · 9 of 40 papers
- SFT · 5 of 40 papers
- IPO · 5 of 40 papers
- PEBBLE · 4 of 40 papers
- PPO · 4 of 40 papers
- SURF · 3 of 40 papers
- RUNE · 3 of 40 papers
- MRN · 3 of 40 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 101 papers; from OpenAlex and the papers' affiliations).
- Google · 10 papers
- University of California, Berkeley · 6 papers
- ByteDance · 6 papers
- Google DeepMind · 5 papers
- NVIDIA · 5 papers
- Carnegie Mellon University · 4 papers
- Georgia Institute of Technology · 4 papers
- Stanford University · 4 papers
- Massachusetts Institute of Technology · 4 papers
- University of Illinois, Urbana Champaign · 4 papers
Use it
Download BibTeX of 33 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Reward modeling versus direct optimization (95 papers)
- Preference-based RL reward learning (26 papers)
Elsewhere in the topic
- RL for LLM reasoning (187 papers)
- Scaling RL with verifiable rewards (119 papers)
- GRPO stability and variants (106 papers)
- RL fine-tuning of generative models (94 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).