Research map / Large language models / Alignment and preferences
LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Maps LLM alignment onto information retrieval's retriever-reranker paradigm and proposes LarPO, an iterative direct preference optimization method using contrastive, LambdaRank, or ListMLE ranking objectives over reward-ranked candidate lists.
Problem
Existing RL-based alignment (PPO, RLHF) is complex, and direct preference optimization methods lack a systematic connection to IR principles; the paper asks how retriever optimization insights (objectives, hard negatives, candidate lists) can improve LLM alignment.
Approach
Treat the LLM as a bi-encoder retriever and the reward model as a cross-encoder reranker, then derive listwise/contrastive preference objectives (contrastive, LambdaRank, ListMLE) from IR ranking assumptions and apply them in iterative on-policy training with temperature-controlled hard negatives and accumulated candidate lists.
Key results
- LarPO (LambdaRank) with LLM-Blender reward reaches 34.9 LC win rate on AlpacaEval 2 for Mistral-Base (7B), versus 15.1 for DPO, per Table 2.
- Listwise objectives (ListMLE, LambdaRank) outperform pairwise DPO on Gemma2-2B-it, e.g. ListMLE LC win rate 49.77 vs 41.39 for pairwise on AlpacaEval 2 (GPT-4 judge).
- Paper reports 38.9% and 13.7% averaged relative improvements on AlpacaEval2 and MixEval-Hard over baselines with the same reward model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LarPO LambdaRank rather than the strongest listed baseline | Mistral-Base (7B) · AlpacaEval 2 | LC WR: 21.5 -> 34.9 | Helped |
| Use LarPO LambdaRank rather than the strongest listed baseline | Mistral-Base (7B) · MixEval-Hard | Score: 0.355 -> 0.452 | Helped |
| Use LarPO Contrastive rather than the strongest listed baseline | Mistral-Instruct (7B) · MixEval-Hard | Score: 0.405 -> 0.418 | Helped |
| Replace pairwise preference optimization with contrastive optimization (ablation) | Mistral-7b-it · MixEval | Score: 0.7175 -> 0.7260 | Helped |
| Use listwise optimization rather than pairwise optimization (ablation) | Mistral-7b-it · MixEval-Hard | Score: 0.4105 -> 0.4400 | Helped |
| Use ListMLE objective rather than pairwise objective (ablation) | Gemma2-2b-it · AlpacaEval 2 | LC Winrate: 41.39 -> 49.77 | Helped |
| Use LambdaRank rather than ListMLE (ablation) | Gemma2-2b-it · MixEval | Score: 0.6715 -> 0.6750 | Mixed |
| Include responses from the previous iteration (ablation) | Gemma2-2b-it iterative preference optimization · AlpacaEval 2 | LC Winrate: 55.06 -> 55.62 | Helped |
| Include responses from all previous iterations (ablation) | Gemma2-2b-it iterative preference optimization · AlpacaEval 2 | Winrate: 66.56 -> 72.50 | Helped |
| Add DPO after SFT (ablation) | Mistral-7b · GSM8K | Greedy Acc: 0.7680 -> 0.7991 | Helped |
| Add DPO after SFT (ablation) | Mistral-7b · Math | Recall@20: 0.5340 -> 0.5416 | Helped |
| Increase candidate list size (ablation) | Iterative preference optimization · AlpacaEval 2 | alignment performance | Helped |
Surprising findings
- Harder negatives generated at lower temperature consistently improve final performance, but very low temperatures reduce diversity and hurt alignment.
- Retaining responses from all previous iterations in the candidate pool improves results over using only the current iteration.
Limitations
- Improvements are reported relative to baselines whose checkpoints come from prior work; the MixEval gains are modest and in some cases the base-model LarPO variants underperform on MixEval.
- Results depend on reward model choice; stronger reward model (FsfairX) gives further gains, so improvements are coupled to reward quality.
- Analysis of SFT and DPO through IR lens is limited to a few math datasets and a single 7B model.
Experimental setup
Datasets: UltraFeedback GSM8K MATH MetaMathQA
Benchmarks: AlpacaEval 2 MixEval MixEval-Hard GSM8K MATH
Metrics: length-controlled win rate win rate MixEval score Recall@N greedy accuracy
Baselines: RRHF SLiC-HF DPO IPO CPO KTO RDPO SimPO Iterative DPO
Builds on: DPO LambdaRank ListMLE iterative DPO SimPO
Closest papers by meaning
- The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.91)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.91)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.90)
- Trustworthy Alignment of Retrieval-Augmented Large Language Models via Reinforcement Learning (ICML 2024 · similarity 0.90)
- Alignment of Large Language Models with Constrained Learning (NeurIPS 2025 · similarity 0.90)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.