Research map / Large language models / Alignment and preferences

MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge

NeurIPS 2026 · Large language models › Alignment and preferences · approach: DPO design choices

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MaPPO, a maximum-a-posteriori extension of DPO that adds a reward-gap-scaled prior term to the MLE preference loss, improving alignment on AlpacaEval 2.0, Arena-Hard, and MT-Bench across several model series and sizes, and as a plugin for DPO variants.

Problem

MLE-based preference optimization (DPO and variants) only maximizes the relative gap between chosen and rejected responses, ignoring absolute reward magnitude; this causes squeezing/confidence degeneration, especially for near-tie pairs.

Approach

Adds a prior over rewards built from an external reward model's gap (Delta_r = r_w - r_l in [0,1]) to the BT likelihood, yielding a MaP loss that scales the rejected-response term by Delta_r while keeping DPO's closed form and adding no hyperparameters; works offline, online (iterative), and as a plugin for SimPO, IPO, CPO.

Key results

• On Mistral-7B-Instruct, MaPPO raises DPO's AlpacaEval 2.0 score from 18.24 to 30.56 and Arena-Hard from 14.2 to 18.4.
• On Qwen2.5-7B-Instruct, MaPPO improves DPO by +6.23 AlpacaEval 2.0 and +13.7 Arena-Hard (45.5 to 59.2).
• As a plugin on Qwen2.5-7B-Instruct, MaPPO improves IPO Arena-Hard by +11.4 and SimPO AlpacaEval by +7.60.

Surprising findings

• MaPPO-trained models are reported to lose a little on MT-Bench for some variants (e.g., SimPO -0.08), which the authors attribute to MT-Bench saturation.

Limitations

• Experiments only up to 32B parameters; larger models not explored.
• Relies on an external reward model as prior; the prior's quality is not varied beyond a few open RMs.
• Gains on MT-Bench are small or slightly negative for some variants, and theoretical analysis (stationary point, Lipschitz) is partly loose.

Experimental setup

Datasets: UltraFeedback

Benchmarks: MT-Bench AlpacaEval 2.0 Arena-Hard v0.1 IFEval GPQA MMLU HellaSwag TruthfulQA GSM8K

Metrics: length-controlled win rate win rate MT-Bench score accuracy

Baselines: IT DPO I-DPO SimPO IPO CPO

Builds on: DPO I-DPO SimPO IPO CPO

Closest papers by meaning

• 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.91)
• Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.91)
• SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024 · similarity 0.91)
• What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.91)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization (ICLR 2025 · similarity 0.90)
• Preference Optimization by Estimating the Ratio of the Data Distribution (NeurIPS 2025 · similarity 0.90)