Research map / Large language models / Alignment and preferences

Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GAPO, a preference-optimization objective that builds a batch-level pessimistic anchor via a first-order adversarial perturbation and down-weights pairs whose margins degrade under it (Anchor Gap), improving alignment and noise robustness over DPO/SimPO-family baselines.

Problem

Margin-based preference optimization treats all pairs with large updates alike, so brittle or noisy pairs (label flips, length bias) can dominate training; it is unclear how to measure whether a pair's preference signal is locally stable enough to trust.

Approach

A batch-conditioned shared adversarial perturbation (gradient-descent direction on the batch-average SimPO margin, radius rho) defines a detached anchor; each pair's Anchor Gap (margin drop under the anchor) sets an instance-wise sigmoid weight on the standard preference gradient, without explicit noise modeling.

Key results

• On Mistral-Instruct 7B, GAPO reaches 35.7 AlpacaEval 2 LC vs 32.1 for SimPO and 26.8 for DPO.
• On Llama-3-Base 8B, GAPO reaches 25.0 AlpacaEval 2 LC vs 22.0 for SimPO.
• Under 20% length-dependent flips on Pythia-2.8B, GAPO reward accuracy is 58.3 vs 49.2 for Dr. DPO and 54.3 for DPO.

Surprising findings

• Explicit noise-robust baseline Dr. DPO falls below plain DPO at flip rates of 20% and above under length-dependent noise.
• The GAPO-weighted gradient norm baseline (GradNorm-Reweight) recovers only part of GAPO's gain, suggesting the alignment term matters.

Limitations

• Overhead of roughly 2x per-step wall-clock relative to SimPO due to anchor gradient and forward pass.
• Noise-enrichment result is from a controlled 20% random-flip setting; enrichment is moderate and most low-weight pairs are still clean.
• Theory is local (small rho, smoothness); no global convergence or robustness guarantee.

Experimental setup

Datasets: UltraFeedback Anthropic HH

Benchmarks: AlpacaEval 2 Arena-Hard v0.1 GSM8k ARC-C

Metrics: length-controlled win rate win rate accuracy reward accuracy

Baselines: DPO IPO KTO CPO ORPO R-DPO SimPO α-DPO Dr. DPO GradNorm-Reweight

Builds on: SimPO DPO Sharpness-Aware Minimization Dr. DPO

Closest papers by meaning

• Distributionally Robust Listwise Preference Optimization (NeurIPS 2026 · similarity 0.90)
• From “Weak” Signals to Strong Models: Preference Delta Aggregation with LoRA Merging (NeurIPS 2026 · similarity 0.89)
• ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization (ICML 2025 · similarity 0.89)
• RePO: Understanding Preference Learning Through ReLU-Based Optimization (NeurIPS 2025 · similarity 0.89)
• Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.89)
• Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.89)