Research map / Reinforcement learning / RL from feedback

Robust Nash Alignment under Preference Uncertainty

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Robust Nash Alignment: a worst-case general-preference game over a Bernoulli-KL ambiguity set of pairwise preference kernels, solved via a four-player primal-dual proxy and single-loop optimistic mirror descent-ascent with O(1/sqrt T) gap guarantees.

Problem

Nash-style preference alignment with a single nominal pairwise preference kernel is brittle when preferences are noisy, heterogeneous, or shift; the robust version over a kernel ambiguity set is non-rectangular and computationally hard.

Approach

Treat the pairwise preference kernel as uncertain within a Bernoulli-KL ball around a nominal kernel, relax the hard constraint via a Lagrangian dual variable into a four-player proxy game, and solve it with optimistic mirror descent-ascent with proven proxy-to-hard gap bounds.</subsection>

Key results

• On IMDb, the robust policy trained at rho_train=0.01 wins against NashMD 98.6% of the time at rho_eval=0.05 and 84.4% at rho_eval=0.50.
• On TL;DR with a non-BT PairRM kernel, the robust policy (Qwen-1.5B, rho_train=0.05) gets win rates of 0.625, 0.615, and 0.732 against DeepSeek-R1, Gemma-4, and Nemotron-3 judges.
• Theory: O(1/sqrt T) average-iterate duality gap for the proxy game, with proxy-to-hard gap bounded by rho*gamma_min.

Surprising findings

• On TL;DR, the robust policy at rho_train=0.01 wins only 49.2% against Nemotron-3 judge, below 50%.

Limitations

• Proxy-to-hard exactness requires an additional dual-optimum inclusion assumption that is not verified in general.
• Robustness gains are evaluated on synthetic kernel perturbations and external LLM judges rather than real preference shift data.
• Summarization experiments use only one fine-tuning model (Qwen-1.5B) in the main text and the rho_train=0.01 case is not uniformly better.

Experimental setup

Datasets: IMDb TL;DR

Benchmarks: IMDb continuation win rate under sampled kernels TL;DR win rate under external LLM judges tabular game robust value

Metrics: win rate worst-case game value duality gap

Baselines: NashMD PairRM-based nominal kernel

Builds on: Nash learning from human feedback Nash-MD optimistic gradient descent ascent distributionally robust optimization

Closest papers by meaning

• Inference Time Nash Alignment (NeurIPS 2026 · similarity 0.89)
• Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning (ICLR 2025 · similarity 0.89)
• Robust Equilibria in Continuous Games: From Strategic to Dynamic Robustness (NeurIPS 2025 · similarity 0.88)
• Self-Play Preference Optimization for Language Model Alignment (ICLR 2025 · similarity 0.88)
• Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.87)
• A Minimaximalist Approach to Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.87)