Research map / Large language models / Alignment and preferences
Active and token-level preference learning
Extend DPO with token-level objectives, reward augmentation, and active preference collection.
58 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2023–2026) · an approach in Alignment and preferences, keyword cluster “preference optimization · dpo · direct preference”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
14 papers in 2023–24 and 44 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 10.7% to 8.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2023 · Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS 2023 · 418 citations)
- 2024 · Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · 154 citations)
- 2025 · Controllable Protein Sequence Generation with LLM Preference Optimization (AAAI 2025 · 3 citations)
- 2026 · Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolation (AAAI 2026 · 3 citations)
Earliest papers here
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS 2023 · 418 citations)
- Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · 154 citations)
- On Softmax Direct Preference Optimization for Recommendation (NeurIPS 2024 · 23 citations)
What has been tried
36 papers here report 415 measured changes (a component added, removed or swapped, compared with a baseline): helped 66%, no clear effect 5%, hurt 24%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Train SD1.5 with PickScore-ranked preference pairs | PartiPrompts · general preference win rate | 59.8% to 63.3% | Helped | Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · 154 citations) |
| add explicit ranking optimization to SFT | Goodreads, LastFM, and MovieLens · HR@1 | Helped | On Softmax Direct Preference Optimization for Recommendation (NeurIPS 2024 · 23 citations) | |
| add general-domain data to medical-only SFT | Medical benchmarks · average score | 75.20 -> 75.90 | Helped | UltraMedical: Building Specialized Generalists in Biomedicine (NeurIPS 2024 · 16 citations) |
| add advantage normalization | APPS Intro. · pass@5 | 18.0% -> 38.1% | Helped | Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (ICML 2024 · 8 citations) |
| add the structural stability metric γ to preference optimization | Six single-attribute datasets (average) · pLDDT | 70.2 -> 73.9 | Helped | Controllable Protein Sequence Generation with LLM Preference Optimization (AAAI 2025 · 3 citations) |
| Add thought prompting without TPO training | AlpacaEval 2 (LC) · win rate (%) | 24.9 -> 17.3 | Hurt | Thinking LLMs: General Instruction Following with Thought Generation (ICML 2025 · 3 citations) |
| Add outlier weighting | 3H static optimization · Norm_Gain | +13.60% -> +14.97% | Helped | Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging (NeurIPS 2025 · 3 citations) |
| add factuality-only DPO rather than instruction-following DPO | Bio · FActScore | 42.3 -> 47.1 | Helped | FLAME : Factuality-Aware Alignment for Large Language Models (NeurIPS 2024 · 3 citations) |
| Add P(True) to AFICE | Commonsense average · accuracy | 0.546 -> 0.606 | Helped | Aligning Large Language Models for Faithful Integrity Against Opposing Argument (AAAI 2025 · 1 citation) |
| Tune alpha to a slightly positive value | AlpacaEval 2.0 · win rate | Helped | AlphaPO: Reward Shape Matters for LLM Alignment (ICML 2025 · 1 citation) | |
| Optimize the BiPO steering vector at the 15th layer rather than other tested layers | AI persona steering · steering effect | Helped | Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization (NeurIPS 2024 · 1 citation) | |
| Remove Negative Category 1 pairs | SafeWorld · factuality | 0.628 -> 0.676 | Helped | SafeWorld: Geo-Diverse Safety Alignment (NeurIPS 2024 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 23 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- UltraFeedback · 9 of 23 papers
- GSM8K · 5 of 23 papers
- HH-RLHF · 3 of 23 papers
- RewardBench · 3 of 23 papers
- UltraChat-200k · 3 of 23 papers
- MMLU · 2 of 23 papers
- MetaMathQA · 2 of 23 papers
- RedPajama · 2 of 23 papers
Baselines
- DPO · 9 of 23 papers
- KTO · 6 of 23 papers
- SimPO · 6 of 23 papers
- IPO · 5 of 23 papers
- SFT · 4 of 23 papers
- SPPO · 3 of 23 papers
- ESM-2 · 2 of 23 papers
- ProGen2 · 2 of 23 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 58 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 6 papers
- The Chinese University of Hong Kong · 4 papers
- Meta · 4 papers
- National University of Singapore · 3 papers
- Google · 3 papers
- University College London · 3 papers
- Amazon · 3 papers
- Tencent · 2 papers
- University of Science and Technology of China · 2 papers
- Shenzhen Research Institute of Big Data · 2 papers
Use it
Download BibTeX of 21 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Robust and calibrated DPO (118 papers)
- DPO design choices (73 papers)
Elsewhere in the topic
- Steering and correcting LLM behavior (127 papers)
- LLM-as-judge and feedback data (111 papers)
- Reward modeling for diverse preferences (90 papers)
- Reward model design and preference models (88 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).