Research map / Large language models / Alignment and preferences
RePO: Understanding Preference Learning Through ReLU-Based Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RePO, a preference optimization objective that replaces SimPO's log-sigmoid weighting with a ReLU hinge on the length-normalized reward margin, filtering pairs whose margin exceeds a threshold gamma, and argues via a convex-envelope analysis that it is sound.
Problem
Preference optimization for LLM alignment relies on mechanisms (sigmoid gradient weighting, SFT regularization) to prevent over-optimization of reward margins; the paper asks whether these are necessary.
Approach
A single ReLU on the length-normalized implicit reward margin minus a target margin gamma yields binary gradient filtering that is the limit of SimPO as beta goes to infinity; gamma acts as the key control over over-optimization and implicit data filtering, and RePO is argued to be the convex envelope of the 0-1 loss.
Key results
- On Llama3-Instruct v0.2 (8B), RePO reaches 57.7 LC win rate on AlpacaEval 2 versus 55.6 for SimPO and 48.2 for DPO.
- On Gemma2-Instruct (9B), RePO reaches 73.6 LC and 59.1 Arena-Hard WR, the best among listed methods.
- Adding RePO's ReLU filtering to DPO raises Gemma2 Arena-Hard from 58.8 to 64.2 (RePO) and 65.7 (RePO++).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the sigmoid-weighted SimPO objective with ReLU threshold filtering | Llama3-Instruct (8B) · AlpacaEval 2 LC | LC win rate: 43.8 -> 46.7 | Helped |
| Replace the sigmoid-weighted SimPO objective with ReLU threshold filtering | Mistral-Instruct (7B) · Arena-Hard | win rate: 23.8 -> 20.3 | Hurt |
| Replace the sigmoid-weighted SimPO objective with ReLU threshold filtering | Llama3-Instruct v0.2 (8B) · AlpacaEval 2 LC | LC win rate: 55.6 -> 57.7 | Helped |
| Replace the sigmoid-weighted SimPO objective with ReLU threshold filtering | Gemma2-Instruct (9B) · Arena-Hard | win rate: 58.8 -> 59.1 | Helped |
| Add RePO filtering to DPO (ablation) | DPO, Llama3-Instruct v0.2 (8B) · AlpacaEval 2 WR | raw win rate: 47.5 -> 51.8 | Helped |
| Add RePO filtering to DPO (ablation) | DPO, Gemma2-Instruct (9B) · Arena-Hard | win rate: 58.8 -> 64.2 | Helped |
| Add RePO++ filtering and weighting to DPO (ablation) | DPO, Gemma2-Instruct (9B) · Arena-Hard | win rate: 58.8 -> 65.7 | Helped |
| Add RePO filtering to SimPO (ablation) | SimPO, Llama3-Instruct v0.2 (8B) · AlpacaEval 2 LC | LC win rate: 55.6 -> 57.7 | Helped |
| Add RePO++ filtering and weighting to SimPO (ablation) | SimPO, Gemma2-Instruct (9B) · Arena-Hard | win rate: 57.8 -> 59.8 | Helped |
| Remove the SFT regularization term (ablation) | SLiC-HF · Preference optimization | performance | No clear effect |
| Use a reference-model margin instead of an explicit target margin (ablation) | DPO with ReLU · Preference optimization | performance | Hurt |
| Increase SimPO beta with fixed gamma (ablation) | SimPO · AlpacaEval 2 and Arena-Hard | win rate | Helped |
Surprising findings
- Removing the log-sigmoid and all regularization still matches or beats SimPO and DPO, with over-optimization controlled only by gamma.
- Directly substituting ReLU into DPO without the target margin gamma causes significant performance degradation, underscoring the role of gamma.
Limitations
- Evaluated only in offline preference learning settings.
- Fixed margin threshold; adaptive or context-aware thresholds not explored.
- Downstream evaluation shows drops on MMLU and GSM8K relative to some baselines.
Experimental setup
Datasets: princeton-nlp/llama3-ultrafeedback-armorm princeton-nlp/gemma2-ultrafeedback-armorm
Benchmarks: AlpacaEval 2 Arena-Hard Huggingface Open LLM Leaderboard (MMLU, ARC, HellaSwag, TruthfulQA, Winograd, GSM8K)
Metrics: length-controlled win rate raw win rate accuracy
Baselines: SFT SLiC-HF DPO IPO CPO KTO ORPO R-DPO SimPO
Builds on: DPO SimPO SLiC-HF RLHF scaling laws
Closest papers by meaning
- SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters (ICLR 2025 · similarity 0.91)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.91)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.91)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.91)
- Group Preference Optimization: Few-Shot Alignment of Large Language Models (ICLR 2024 · similarity 0.90)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.