Research map / Large language models / Alignment and preferences

Theoretical Limits of Language Model Alignment

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives the exact optimal reward-gain-vs-KL limit for KL-regularized alignment (gain = λ·Jeffreys divergence), gives a base-model-only covariance estimator, and shows best-of-N tracks the frontier while PPO/GRPO fall short.

Problem

The theoretical limit of reward gain achievable for a given KL budget in KL-regularized LM alignment is unknown, and prior bounds were non-achievable upper bounds (square-root-KL) that depend on the aligned model, so practitioners cannot tell how close methods are to optimal.

Approach

Shows the optimal expected reward gain under KL-regularized alignment equals λ times the Jeffreys divergence between base and aligned model, rewrites it as a covariance under the base model alone (estimable from base samples before training), and extends the analysis to proxy rewards and reward ensembling.

Key results

• Optimal reward gain under KL-regularized alignment is exactly λ·JD(base||aligned), and the estimator converges with typically fewer than 1,000 base samples per query.
• Best-of-N closely tracks the derived theoretical reward-KL Pareto frontier across the full KL range on both BeaverTails (Zephyr-7B) and TL;DR (Pythia-1B).
• PPO and GRPO remain substantially below the theoretical frontier at every KL budget tested, recovering only a fraction of the reward gain achieved by best-of-N.

Surprising findings

• Best-of-N, an inference-time method, is nearly Pareto optimal across the whole reward-KL trade-off, while tuned PPO and GRPO are far from the frontier.
• The sqrt(KL) scaling previously conjectured as the reward-gain law is replaced by an exact Jeffreys-divergence characterization.

Experimental setup

Datasets: BeaverTails Reddit TL;DR PKU-SafeRLHF UltraChat

Benchmarks: BeaverTails Reddit TL;DR

Metrics: expected reward gain KL divergence reward-KL Pareto frontier

Baselines: PPO GRPO best-of-N

Builds on: PPO GRPO Direct Preference Optimization Best-of-N sampling

Closest papers by meaning

• RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.92)
• Alignment of Large Language Models with Constrained Learning (NeurIPS 2025 · similarity 0.92)
• Noise Contrastive Alignment of Language Models with Explicit Rewards (NeurIPS 2024 · similarity 0.91)
• Inference-Time Reward Hacking in Large Language Models (NeurIPS 2025 · similarity 0.91)
• Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization (ICLR 2025 · similarity 0.91)
• Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.91)