Research map / Reinforcement learning / RL from feedback
Preference-based RL reward learning
Learn rewards from pairwise preferences with improved queries, exploration, and reward-agnostic methods.
26 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in RL from feedback, keyword cluster “human · rlhf · preferences”
State History Tried Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
10 papers in 2023–24 and 12 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 11.0% to 1.9%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Model-Free Preference-Based Reinforcement Learning (AAAI 2016 · 64 citations)
- 2020 · Preference-based Reinforcement Learning with Finite-Time Guarantees (NeurIPS 2020 · 8 citations)
- 2021 · B-Pref: Benchmarking Preference-Based Reinforcement Learning (NeurIPS 2021 · 3 citations)
- 2022 · Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning (NeurIPS 2022 · 1 citation)
- 2023 · Direct Preference-based Policy Optimization without Reward Modeling (NeurIPS 2023 · 5 citations)
- 2024 · Can You Rely on Synthetic Labellers in Preference-Based Reinforcement Learning? It’s Complicated (AAAI 2024 · 3 citations)
- 2025 · CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries (ICML 2025 · 0 citations)
- 2026 · Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning (ICML 2026 · 0 citations)
Earliest papers here
- Model-Free Preference-Based Reinforcement Learning (AAAI 2016 · 64 citations)
- Preference-based Reinforcement Learning with Finite-Time Guarantees (NeurIPS 2020 · 8 citations)
- B-Pref: Benchmarking Preference-Based Reinforcement Learning (NeurIPS 2021 · 3 citations)
What has been tried
14 papers here report 116 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 10%, hurt 14%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| select samples with highest inter-annotator agreement instead of random samples | stand and walk · normalized returns | stand: 0.66 -> 0.55; walk: 0.87 -> 0.56 | Hurt | Can You Rely on Synthetic Labellers in Preference-Based Reinforcement Learning? It’s Complicated (AAAI 2024 · 3 citations) |
| Add the ambiguity loss to the model without either contrastive loss | dial-turn, skip rate 0.5 · success rate | 63.20 ± 4.79 -> 69.00 ± 11.20 | Helped | CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries (ICML 2025 · 0 citations) |
| Use timestep as the stage approximation and Interval Span Ratio for segment distance | Ablation study on benchmark tasks · task performance | Hurt | STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning (NeurIPS 2025 · 0 citations) | |
| Use uniform importance for return redistribution instead of forward-dynamics attention | MetaWorld and DMC · policy performance | Hurt | Hindsight PRIORs for Reward Learning from Human Preferences (ICLR 2024 · 0 citations) | |
| Add policy-aligned query selection | Walker_walk, Cheetah_run, Walker_run · task performance | Helped | Query-Policy Misalignment in Preference-Based Reinforcement Learning (ICLR 2024 · 0 citations) | |
| Add warm start to the reward model | Walker-walk, ε=0.3 · episode return | 457 ± 190 -> 741 ± 139 | Helped | RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences (ICML 2024 · 0 citations) |
| Add RLT construction to MR using the exponential score function | button-press-topdown, box-close, dial-turn and lever-pull medium-replay, 500 feedbacks · success rate | 9.60 -> 12.87; 10.33 -> 22.73; 50.20 -> 65.87; 50.53 -> 57.87 | Helped | Listwise Reward Estimation for Offline Preference-based Reinforcement Learning (ICML 2024 · 0 citations) |
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 22 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 4 papers
- Chinese Academy of Sciences · 4 papers
- Apple · 2 papers
- Nanjing University · 2 papers
- Princeton University · 2 papers
- Genentech · 2 papers
- Cornell University · 2 papers
- Seoul National University · 2 papers
- Tencent · 2 papers
Use it
Download BibTeX of 25 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Theory of RLHF (110 papers)
- Reward modeling versus direct optimization (95 papers)
Elsewhere in the topic
- RL for LLM reasoning (187 papers)
- Scaling RL with verifiable rewards (119 papers)
- GRPO stability and variants (106 papers)
- RL fine-tuning of generative models (94 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).