Research map / Large language models / Alignment and preferences

Distributionally Robust Listwise Preference Optimization

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a pointwise total-variation robust Plackett-Luce listwise preference objective whose worst-case ranking is found by ascending-score sorting, with offline convergence and online weak-convexity guarantees and experiments on UltraFeedback with Qwen3 models.

Problem

Preference optimization for LLM alignment with listwise rankings is sensitive to ranking-label noise (annotator inconsistency, near-ties, reward-model errors), which existing robust DPO methods, focused on pairwise data or prompt/pair distributions, do not address.

Approach

Robustify the ranking label conditional on a realized candidate list with a pointwise TV ambiguity set; the robust PL loss decomposes exactly into (1-rho) nominal PL plus rho times a worst-case PL loss, and the worst-case ranking is the ascending-score sort, giving O(K log K) evaluation.

Key results

• Offline, Qwen3-8B under top-rank corruption at epsilon=1.0: Robust PL (rho=0.05) reaches Kendall tau 0.146 vs 0.103 for Nominal PL and 0.039 for Nominal BT.
• Online Qwen3-8B: robust K=4 (rho=0.05) achieves +610.7 Delta Reward and 28.9% GPT-4 win vs chosen, versus +467.0 and 23.8% for non-robust K=4.
• Clean-label offline performance is slightly reduced: Qwen3-8B Robust PL (rho=0.05) 0.347 vs Nominal PL 0.362 Kendall tau.

Surprising findings

• Non-robust listwise K=4 does not consistently improve over binary K=2 in online alignment, suggesting larger lists add reward-model ranking noise.
• Robust PL with rho=0.02 on Qwen3-0.6B clean-label offline performance is slightly below nominal, and TV-DR-DPO/KLDPO underperform nominal BT under some noise conditions.

Limitations

• Robustness radius rho must be tuned; over-regularization at larger rho reduces performance.
• Theory assumes log-linear policy class and bounded features, which is a simplification for LLMs.
• Experiments are limited to UltraFeedback with K=4 and Qwen models up to 8B.

Experimental setup

Datasets: UltraFeedback

Benchmarks: UltraFeedback clean held-out rankings U10 held-out evaluation set

Metrics: Kendall's tau reward-model gain (Delta Reward) reward-model win rate vs SFT GPT-4 win rate vs chosen Top-1 Pairwise agreement

Baselines: Nominal BT (DPO) TV-DR-DPO KLDPO Nominal PL SAIL (binary, K=2)

Builds on: DPO Plackett-Luce model SAIL TV-DR-DPO KLDPO

Closest papers by meaning

• Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.91)
• Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.90)
• Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.90)
• Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment (NeurIPS 2026 · similarity 0.90)
• ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.89)
• TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.89)