Research map / Large language models / Alignment and preferences
Reward model design and preference models
Combine, actively label, and generalize reward and preference models beyond Bradley-Terry.
88 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Alignment and preferences, keyword cluster “feedback · human · rlhf”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
15 papers in 2023–24 and 70 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 11.5% to 13.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Coactive Critiquing: Elicitation of Preferences and Features (AAAI 2017 · 19 citations)
- 2018 · An Axiomatization of the Eigenvector and Katz Centralities (AAAI 2018 · 13 citations)
- 2019 · Reward Constrained Interactive Recommendation with Natural Language Feedback (NeurIPS 2019 · 5 citations)
- 2023 · Personalized Reward Learning with Interaction-Grounded Learning (IGL) (ICLR 2023 · 0 citations)
- 2024 · Eureka: Human-Level Reward Design via Coding Large Language Models (ICLR 2024 · 48 citations)
- 2025 · VTON-VLLM: Aligning Virtual Try-On Models with Human Preferences (NeurIPS 2025 · 5 citations)
- 2026 · SparseRM: A Lightweight Preference Modeling with Sparse Autoencoder (AAAI 2026 · 1 citation)
Earliest papers here
- Coactive Critiquing: Elicitation of Preferences and Features (AAAI 2017 · 19 citations)
- An Axiomatization of the Eigenvector and Katz Centralities (AAAI 2018 · 13 citations)
- Reward Constrained Interactive Recommendation with Natural Language Feedback (NeurIPS 2019 · 5 citations)
What has been tried
45 papers here report 478 measured changes (a component added, removed or swapped, compared with a baseline): helped 63%, no clear effect 7%, hurt 26%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add iterative evolutionary reward search rather than spending the same sample budget on initial samples | Isaac and Dexterity · task performance | Helped | Eureka: Human-Level Reward Design via Coding Large Language Models (ICLR 2024 · 48 citations) | |
| Initialize Scaled BT with Helpfulness-Only Regression and apply ExPO | RewardBench · overall accuracy | 93.7 -> 94.1 | Helped | HelpSteer2-Preference: Complementing Ratings with Preferences (ICLR 2025 · 4 citations) |
| Replace Daring Anteater with Open Assistant for SFT | MT Bench · GPT-4-Turbo score | 7.96 -> 6.75 | Hurt | HelpSteer 2: Open-source dataset for training top-performing reward models (NeurIPS 2024 · 4 citations) |
| Add training on negated relation comparisons | Summarize from Feedback · s_neg | 60.8 -> 87.9 | Helped | Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models (ICML 2025 · 3 citations) |
| add entity-facet embeddings Xe | REDIAL · Distinct-2 | 0.118 -> 0.126 | Helped | FacetCRS: Multi-Faceted Preference Learning for Pricking Filter Bubbles in Conversational Recommender System (AAAI 2024 · 3 citations) |
| Increase batch size | AlpacaEval (Llama3-70B-Instruct judge) · score | Helped | Distributional Preference Alignment of LLMs via Optimal Transport (NeurIPS 2024 · 3 citations) | |
| Use all subsets instead of General + STEM + Code | RM-Bench · accuracy | 79.9 -> 78.5 | Hurt | HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages (NeurIPS 2025 · 2 citations) |
| Add observation loss | TARA single-tool · accuracy (Avg.) | 90.23 -> 94.23 | Helped | Tool-Augmented Reward Modeling (ICLR 2024 · 2 citations) |
| apply the learned preference flow iteratively for two iterations | MNIST conditional image generation · preference score | 0.9841 -> 0.9996 | Helped | Preference Alignment with Flow Matching (NeurIPS 2024 · 2 citations) |
| Remove safety-direction SMV projection from LEGEND | Safe-RLHF · accuracy gain over Origin | -1.04 (± 2.42) | Hurt | LEGEND: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets (AAAI 2025 · 1 citation) |
| Replace preference weight sampling with distribution weight sampling | OpenStreetMap Routing and HomeGrid · overall performance | Hurt | MAPLE: A Framework for Active Preference Learning Guided by Large Language Models (AAAI 2025 · 1 citation) | |
| remove the two-stage/CNF-based synthesis and auxiliary-concept handling, using one-step synthesis | CONTINGENCY full dataset · mIOU (%) | 76.11 -> 60.74 | Hurt | SYNAPSE: SYmbolic Neural-Aided Preference Synthesis Engine (AAAI 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 37 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- UltraFeedback · 5 of 37 papers
- RewardBench · 3 of 37 papers
- HelpSteer2 · 3 of 37 papers
- HH-RLHF · 3 of 37 papers
- Safe-RLHF · 2 of 37 papers
- Alpaca · 2 of 37 papers
- SummEval · 2 of 37 papers
- TriviaQA · 2 of 37 papers
Baselines
- Skywork-Reward-Gemma-2-27B · 5 of 37 papers
- DPO · 3 of 37 papers
- Llama-3.1-Nemotron-70B-Reward · 3 of 37 papers
- IPO · 2 of 37 papers
- SimPO · 2 of 37 papers
- GPT-4o · 2 of 37 papers
- BERTScore · 2 of 37 papers
- Skywork-Reward-Llama-3.1-8B · 2 of 37 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 85 papers; from OpenAlex and the papers' affiliations).
- Google DeepMind · 5 papers
- Google · 5 papers
- NVIDIA · 4 papers
- Harvard University · 3 papers
- Microsoft · 3 papers
- Massachusetts Institute of Technology · 3 papers
- Hong Kong University of Science and Technology · 3 papers
- Amazon · 3 papers
- University of Cambridge · 3 papers
- California Institute of Technology · 2 papers
Use it
Download BibTeX of 23 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Steering and correcting LLM behavior (127 papers)
- Reward modeling for diverse preferences (90 papers)
- LLM-as-judge and feedback data (111 papers)
Elsewhere in the topic
- Robust and calibrated DPO (118 papers)
- DPO design choices (73 papers)
- Active and token-level preference learning (58 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).