Research map / Large language models / Alignment and preferences
SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SimPER, a hyperparameter-free and reference-free offline preference optimization objective that minimizes the reverse perplexity (inverse perplexity) of chosen responses while maximizing it for rejected ones.
Problem
Existing offline preference optimization methods (DPO, IPO, KTO, SLiC, CPO, SimPO) require hyperparameters that must be extensively tuned and often a reference model, increasing alignment cost and sensitivity.
Approach
Optimize the inverse perplexity (exponentiated length-averaged log-likelihood) of chosen versus rejected responses directly as the loss, giving an objective with no hyperparameters and no reference model; the authors argue it balances gradients between chosen and rejected responses and approximately minimizes total variation distance.
Key results
- On AlpacaEval 2, SimPER reaches 25.2 LC win rate on Llama3-8B-Base versus 22.0 for SimPO, and 48.5 on Llama3-8B-Instruct versus 44.7 for SimPO.
- On Open LLM Leaderboard, SimPER achieves the best average rank of 1.3 on Llama3-8B-Base, compared with 3.4 for SimPO.
- SimPER outperforms SimPO by 19.48 points on GSM8K and 4.23 points on IFEval on Llama3-Base, per the text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SimPER instead of SimPO | Mistral-7B-Base · AlpacaEval 2 LC (%) | LC win rate: 21.5 -> 22.4 | Helped |
| Use SimPER instead of SimPO | Mistral-7B-Instruct · AlpacaEval 2 WR (%) | raw win rate: 34.8 -> 39.5 | Helped |
| Use SimPER instead of SimPO | Llama3-8B-Base · MT-Bench | GPT-4 score: 7.7 -> 7.7 | No clear effect |
| Use SimPER instead of SimPO | Llama3-8B-Instruct · AlpacaEval 2 WR (%) | raw win rate: 40.5 -> 45.7 | Helped |
| Use SimPER instead of SimPO | Llama3-8B-Base · IFEval | score: 37.55 -> 41.78 | Helped |
| Use SimPER instead of SimPO | Llama3-8B-Base · GSM8K | score: 31.54 -> 51.02 | Helped |
| add length normalization (ablation) | SimPER on Mistral-7B-Base · IFEval | score: 1.66 -> 15.83 | Helped |
| add length normalization (ablation) | SimPER on Mistral-7B-Base · MATH | score: 3.25 -> 2.57 | Hurt |
| add length normalization (ablation) | SimPER on Llama3-8B-Base · GSM8K | score: 44.28 -> 51.02 | Helped |
| add length normalization (ablation) | SimPER on Llama3-8B-Base · Winograd | score: 76.64 -> 76.24 | Hurt |
| incorporate a reference model (ablation) | SimPER on Mistral-7B-Base · IFEval | score: 15.83 -> 16.32 | Helped |
| incorporate a reference model (ablation) | SimPER on Mistral-7B-Base · MATH | score: 2.57 -> 2.11 | Hurt |
Surprising findings
- MT-Bench shows poor separability across methods, with most scores falling within a narrow range.
- Adding a reference model to SimPER generally decreases downstream performance in the ablation.
Limitations
- MT-Bench has limited separability across methods, making it a weak discriminator.
- The TVD theoretical link is derived with length averaging removed, so it only loosely covers the practical objective.
- Baselines were tuned with hyperparameter search, while the method uses a learning-rate search only, so comparisons depend on the tuning budget.
Experimental setup
Datasets: UltraFeedback Binarized Anthropic-HH
Benchmarks: AlpacaEval 2 MT-Bench Open LLM Leaderboard v1 Open LLM Leaderboard v2 Anthropic-HH
Metrics: length-controlled win rate raw win rate MT-Bench GPT-4 score accuracy win rate vs chosen responses average rank Spearman correlation
Baselines: DPO SLiC IPO KTO CPO SimPO
Builds on: DPO SimPO IPO KTO SLiC
Closest papers by meaning
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.93)
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.92)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.92)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.92)
- MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models (NeurIPS 2024 · similarity 0.91)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.