Research map / Large language models / Alignment and preferences
Reward modeling for diverse preferences
Improve RLHF reward models with language feedback, diverse preferences, and error analysis.
90 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Alignment and preferences, keyword cluster “feedback · human · rlhf”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
27 papers in 2023–24 and 62 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 20.6% to 11.9%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Teaching Machines to Describe Images with Natural Language Feedback (NeurIPS 2017 · 38 citations)
- 2023 · Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (NeurIPS 2023 · 34 citations)
- 2024 · RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · 69 citations)
- 2025 · Self-Play Preference Optimization for Language Model Alignment (ICLR 2025 · 11 citations)
- 2026 · Expanding the Capabilities of Reinforcement Learning via Text Feedback (ICML 2026 · 0 citations)
Earliest papers here
- Teaching Machines to Describe Images with Natural Language Feedback (NeurIPS 2017 · 38 citations)
- Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (NeurIPS 2023 · 34 citations)
- RRHF: Rank Responses to Align Language Models with Human Feedback (NeurIPS 2023 · 15 citations)
What has been tried
62 papers here report 676 measured changes (a component added, removed or swapped, compared with a baseline): helped 63%, no clear effect 9%, hurt 22%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add chain-of-thought reasoning to the base zero-shot prompt | Summarization, helpful dialogue, and harmless dialogue · AI labeler alignment | 76.1% -> 77.5%; 67.8% -> 69.1%; 69.4% -> 70.6% | Helped | RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · 69 citations) |
| Add the SFT loss to PRO | HH-RLHF raw (Total) · BLEU | 9.85 -> 21.54 | Helped | Preference Ranking Optimization for Human Alignment (AAAI 2024 · 57 citations) |
| Add natural language feedback | TL;DR summarization · human preference win rate (%) | 10.6 -> 15.1 | Helped | Chain of Hindsight aligns Language Models with Feedback (ICLR 2024 · 27 citations) |
| Decouple helpfulness and harmlessness annotation and training | Preference annotation · Inter-Rater Agreement Rate | Helpfulness: 69.00% and Safety: 66.53% compared to 61.65% | Helped | Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · 22 citations) |
| Use K=5 rather than K=2 when estimating the win rate | AlpacaEval 2.0 · win rate | Mixed | Self-Play Preference Optimization for Language Model Alignment (ICLR 2025 · 11 citations) | |
| add Evaluation Fine-Tuning (EFT) data | Open Assistant reward evaluation · pairwise accuracy | 65.1% -> 78.7% | Helped | Self-Rewarding Language Models (ICML 2024 · 10 citations) |
| Increase ultrafeedback prompt count from 10k to 40k | AlpacaEval · win rate | 94.29% -> 93.11% | Hurt | ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models (ICML 2024 · 7 citations) |
| Set IB dimensionality to 128 and β to 0.1 | Anthropic-Harmless · win rate | 128; 0.1 | Helped | InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling (NeurIPS 2024 · 5 citations) |
| Use prompt-implied preference labels instead of post-hoc rescoring | Harmlessness prompts · GPT-4 preferred (%) | 86.0 / 14.0 | Helped | RLCD: Reinforcement Learning from Contrastive Distillation for LM Alignment (ICLR 2024 · 4 citations) |
| Use confidence-query prompts in PPO but retain the vanilla reward model | GSM8K (DA) · ECE | 0.8843 -> 0.8954 | Hurt | Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · 3 citations) |
| Use FedBis with U=3 instead of FedBiscuit | TL;DR summarization · win rate | 77.34% -> 83.81% | Helped | Towards Federated RLHF with Aggregated Client Preference for LLMs (ICLR 2025 · 3 citations) |
| Combine mutation and crossover rather than using crossover only or mutation only | Autonomous Driving · fitness | Helped | REvolve: Reward Evolution with Large Language Models using Human Feedback (ICLR 2025 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 42 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- UltraFeedback · 6 of 42 papers
- RewardBench · 5 of 42 papers
- Anthropic/hh-rlhf · 4 of 42 papers
- HH-RLHF · 4 of 42 papers
- AlpacaEval · 4 of 42 papers
- GSM8K · 4 of 42 papers
- XSum · 3 of 42 papers
- Alpaca · 3 of 42 papers
Baselines
- DPO · 10 of 42 papers
- PPO · 4 of 42 papers
- RLOO · 3 of 42 papers
- SPPO · 3 of 42 papers
- SimPO · 3 of 42 papers
- SFT · 3 of 42 papers
- IPO · 3 of 42 papers
- Eureka · 2 of 42 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 89 papers; from OpenAlex and the papers' affiliations).
- Meta · 7 papers
- Google DeepMind · 7 papers
- University of California, Berkeley · 6 papers
- Google · 5 papers
- Microsoft · 5 papers
- OpenAI · 4 papers
- Alibaba Group · 3 papers
- Peking University · 3 papers
- Harvard University · 3 papers
- Tencent · 3 papers
Use it
Download BibTeX of 24 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Steering and correcting LLM behavior (127 papers)
- LLM-as-judge and feedback data (111 papers)
- Reward model design and preference models (88 papers)
Elsewhere in the topic
- Robust and calibrated DPO (118 papers)
- DPO design choices (73 papers)
- Active and token-level preference learning (58 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).