Research map / Large language models / Alignment and preferences

Inference Time Nash Alignment

NeurIPS 2026 · Large language models › Alignment and preferences · approach: DPO design choices · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces inference-time alignment under general (non-Bradley-Terry) preferences by computing a Nash equilibrium over N sampled responses, with two training-free methods (Best-of-Nash via LP and Nash Mirror Descent) that achieve a duality gap matching an Omega(eps*C_uni) lower bound.

Problem

Inference-time alignment with black-box models relies on scalar Bradley-Terry reward models, which cannot represent general (intransitive or group-level) preferences and are prone to reward hacking under Best-of-N.

Approach

Casts inference-time alignment as finding a Nash equilibrium of a zero-sum game between policies over N sampled candidates, using pairwise preference queries; proposes an LP-based Best-of-Nash and a self-play Nash Mirror Descent, with matching upper and lower duality-gap bounds.

Key results

• On TLDR with LLaMA3-SFT and N=64, BoN and NMD raise expected win rate from 62.9% to 73.5% and 73.1%, matching the LLaMA3-DPO comparison.
• On HelpSteer2 and UltraFeedback, BoN reaches 68.8% and 48.8% vs base 44.1% and 23.9%; NMD reaches 67.8% and 46.7%.
• On HelpSteer2 with Qwen3-0.6B, NMD (52.0%) and Best-of-Nash (50.5%) outperform fine-tuned Nash-MD-PG (41.5%) and base (34.2%) without parameter updates.

Surprising findings

• A simple Borda best-of-N baseline attains win rates close to Best-of-Nash and NMD; 73.8% of N=64 subsamples have a Condorcet winner, making the equilibrium trivially pure.
• Fine-tuned Nash-MD's gain comes mostly from converting losses into draws rather than increasing outright wins.

Limitations

• Guarantees depend on preference-oracle error epsilon, so biases of the preference model are inherited and out-of-distribution prompts are a risk.
• Inference-time alignment can only reweight responses reachable under the base policy; neither method can recover negligible-probability responses.
• Per-prompt analysis ignores shared structure across prompts; O(N^2) pairwise queries are costly.

Experimental setup

Datasets: TLDR HelpSteer2 UltraFeedback

Benchmarks: TLDR HelpSteer2 UltraFeedback

Metrics: expected win rate (win + draw/2) win/draw/lose rate post-query run-time

Baselines: Best-of-N (Bradley-Terry reward) Borda Best-of-N Nash-MD-PG LLaMA3-DPO base SFT policy

Builds on: Nash learning from human feedback Best-of-N sampling optimistic mirror descent rejection sampling

Closest papers by meaning

• Robust Nash Alignment under Preference Uncertainty (NeurIPS 2026 · similarity 0.89)
• Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning (ICLR 2025 · similarity 0.88)
• Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment (ICLR 2025 · similarity 0.86)
• Self-Play Preference Optimization for Language Model Alignment (ICLR 2025 · similarity 0.86)
• As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss (ICLR 2025 · similarity 0.86)
• Human Alignment of Large Language Models through Online Preference Optimisation (ICML 2024 · similarity 0.86)