Research map / Large language models / Alignment and preferences
Steering and correcting LLM behavior
Align LLMs through representation editing, influence functions, and intrinsic corpus signals.
127 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2022–2026) · an approach in Alignment and preferences, keyword cluster “feedback · human · rlhf”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 1 of 1 tracked benchmark.
28 papers in 2023–24 and 97 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 21.4% to 18.6%.
Open questions the data points to
- MT-Bench (score): best went from 7.74 (up to 2024) to 8.91 in 2025 — Improving Model Alignment Through Collective Intelligence of Open-Source Models (ICML 2025 · 0 citations)
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2022 · Fine-tuning language models to find agreement among humans with diverse preferences (NeurIPS 2022 · 110 citations)
- 2023 · OpenAssistant Conversations - Democratizing Large Language Model Alignment (NeurIPS 2023 · 130 citations)
- 2024 · Fundamental Limitations of Alignment in Large Language Models (ICML 2024 · 45 citations)
- 2025 · Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (ICLR 2025 · 6 citations)
- 2026 · Safety Alignment of Large Language Models via Contrasting Safe and Harmful Distributions (AAAI 2026 · 2 citations)
Earliest papers here
- Fine-tuning language models to find agreement among humans with diverse preferences (NeurIPS 2022 · 110 citations)
- Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits (NeurIPS 2022 · 12 citations)
- OpenAssistant Conversations - Democratizing Large Language Model Alignment (NeurIPS 2023 · 130 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MT-Bench · score (higher is better)
4 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 7.74 | Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · 4 citations) |
| 2025 | 8.91 | Improving Model Alignment Through Collective Intelligence of Open-Source Models (ICML 2025 · 0 citations) |
What has been tried
72 papers here report 761 measured changes (a component added, removed or swapped, compared with a baseline): helped 70%, no clear effect 5%, hurt 17%, mixed 8%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add random mismatch loss to GA | TruthfulQA normal prompts · utility reward | -3.838 -> -2.982 | Helped | Large Language Model Unlearning (NeurIPS 2024 · 34 citations) |
| add a reference answer, rationale generation, and score rubric | Human-model evaluation · correlation | 0.516; 0.634; 0.646 | Helped | FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets (ICLR 2024 · 9 citations) |
| Remove similarity update | AdvBench, GCG white-box attack · attack success rate | 19% -> 22% | Hurt | RAIN: Your Language Models Can Align Themselves without Finetuning (ICLR 2024 · 7 citations) |
| Filtering the Magpie-Pro 300K data (quality, difficulty, reward filters, longest responses) | AlpacaEval 2 vs GPT-4-Turbo · LC (%) | 21.65 -> 25.08 LC | Helped | Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (ICLR 2025 · 6 citations) |
| Use LoRA training | HarmBench · average ASR | 40.74 -> 36.95 | Helped | Mission Impossible: A Statistical Perspective on Jailbreaking LLMs (NeurIPS 2024 · 6 citations) |
| Add harmless data to training | HH-RLHF · win rate vs. base model | 89.3% | Helped | Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation (ICML 2024 · 4 citations) |
| Add chain-of-thought reasoning | TL;DR train principles · win rate vs. Llama3-70B | 60% -> 68% | Helped | Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels (NeurIPS 2024 · 4 citations) |
| Apply DPO to Janus* 7B | Multifaceted Bench · average score | 4.17 -> 4.24 | Helped | Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · 4 citations) |
| Add reward weighting to the TQR architecture | Anthropic-Harmless, Anthropic-Helpful, and OpenAI-Summary · reward accuracy | 70.06 -> 70.27; 70.73 -> 72.30; 69.81 -> 72.37; 69.32 -> 74.84; 60.55 -> 61.79; 58.89 -> 64.31 | Helped | Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment (AAAI 2025 · 3 citations) |
| Train on best-of-n samples only, without the IPO-BoN term | Anthropic HH and OpenAI TL;DR · win rate | Hurt | BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling (NeurIPS 2024 · 3 citations) | |
| Reverse which action token means Cooperate versus Defect during Game-reward training | IPD vs TFT · defection at training endpoint | nearly 100% | No clear effect | Moral Alignment for LLM Agents (ICLR 2025 · 2 citations) |
| Use identity-mapping warm-up before Q-A-C training | BeaverTails and HarmfulQA · Performance | Helped | Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 56 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- UltraFeedback · 10 of 56 papers
- HH-RLHF · 7 of 56 papers
- MATH · 4 of 56 papers
- UltraChat · 4 of 56 papers
- GSM8K · 4 of 56 papers
- HelpSteer · 3 of 56 papers
- PKU-SafeRLHF · 3 of 56 papers
- BeaverTails · 3 of 56 papers
Baselines
- DPO · 12 of 56 papers
- SFT · 6 of 56 papers
- PPO · 6 of 56 papers
- GPT-4o · 6 of 56 papers
- Claude 3.5 Sonnet · 3 of 56 papers
- MOD · 3 of 56 papers
- KTO · 3 of 56 papers
- Best-of-N · 3 of 56 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 125 papers; from OpenAlex and the papers' affiliations).
- Peking University · 5 papers
- University of Washington · 4 papers
- IBM · 4 papers
- Meta · 4 papers
- Carnegie Mellon University · 4 papers
- University of California, Berkeley · 4 papers
- Harbin Institute of Technology · 3 papers
- Apple · 3 papers
- Microsoft · 3 papers
- Korea Advanced Institute of Science and Technology · 3 papers
Use it
Download BibTeX of 26 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Reward modeling for diverse preferences (90 papers)
- LLM-as-judge and feedback data (111 papers)
- Reward model design and preference models (88 papers)
Elsewhere in the topic
- Robust and calibrated DPO (118 papers)
- DPO design choices (73 papers)
- Active and token-level preference learning (58 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).