Research map / Large language models / Alignment and preferences

AMPO: Active Multi Preference Optimization for Self-play Preference Selection

ICML 2025 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AMPO, an on-policy multi-preference method that picks one top-reward positive and K active negatives (BottomK, k-means Coreset, or Opt-Select coverage) and trains with the REFA group-contrastive loss, reporting state-of-the-art AlpacaEval 2 results on Llama-3-8B-Instruct and Mistral-7B.

Problem

Multi-preference alignment with self-play generates many candidate responses per query, and including all of them in a group-contrastive objective is computationally infeasible and redundant; the paper asks how to select a small informative subset.

Approach

Combine on-policy sampling of N=32 responses, a REFA group-contrastive loss with one best positive and K active negatives, and active subset selection (bottom-K by reward, reward-weighted k-means coreset, or a weighted k-medoids-style Opt-Select coverage objective) motivated by a Lipschitz suppression argument.

Key results

• On Llama-3-Instruct (8B), AMPO-Coreset reaches 52.4% LC and 52.1% WR on AlpacaEval 2, versus 47.6% LC and 44.7% WR for SimPO.
• On Mistral-Instruct (7B), AMPO-Opt-Select reaches 33.1% LC on AlpacaEval 2 versus 30.1% for SimPO.
• AMPO-Opt-Select with 1-vs-7 negatives achieves 46.4 on Arena-Hard, and performance declines when fewer negatives (1-vs-3) are used (49.6 LC).

Measured effects reported in the paper

Surprising findings

• Using a single positive (1-vs-k) outperformed multiple positives (k-vs-k) in the reported ablations, and the 4-vs-4 setting was worse on LC win rate.
• Weighting pairs by reward gap (l1) improved AlpacaEval but reduced Arena-Hard and MT-Bench scores relative to uniform weighting (l0).

Limitations

• The theoretical guarantees rely on a Lipschitz assumption on the policy that the authors acknowledge is rarely known exactly.
• Results are reported on a small set of base models (Llama-3-8B and Mistral-7B) and largely on AlpacaEval 2, with hyperparameters (beta, gamma) tuned by the authors.
• The MIP and local search formulation is only approximate for large N; the reported 5x local search bound is stated for weighted k-medoids, not verified end-to-end.

Experimental setup

Datasets: UltraFeedback

Benchmarks: AlpacaEval 2 Arena-Hard v0.1 MT-Bench

Metrics: length-controlled win rate win rate GPT-4 score

Baselines: SimPO DPO IPO CPO KTO ORPO R-DPO RRHF SLiC-HF

Builds on: REFA SimPO InfoNCA DPO

Closest papers by meaning

• Self-Play Preference Optimization for Language Model Alignment (ICLR 2025 · similarity 0.92)
• AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.90)
• Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.90)
• Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.90)
• Group Preference Optimization: Few-Shot Alignment of Large Language Models (ICLR 2024 · similarity 0.89)
• Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.89)