Research map / Reinforcement learning / RL from feedback
Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes χPO, a one-line modification of DPO that replaces the log link with z + log z, implicitly implementing chi-squared-regularized pessimism to get single-policy concentrability guarantees against overoptimization in offline alignment.
Problem
Overoptimization (reward hacking) in offline language model alignment, where KL-regularized methods like DPO degrade as they overfit an offline reward model under limited data coverage.
Approach
Replace the logarithmic link in the DPO objective with phi(z)=z+log z, which corresponds to mixed chi-squared plus KL regularization; this implicitly applies pessimism via chi-squared divergence, yielding sample complexity bounds that scale with single-policy concentrability.
Key results
- On TL;DR with beta=0.005 and 4 epochs, chiPO achieves 51.6% winrate while DPO drops to 0.5%.
- At beta=0.05 and 4 epochs, chiPO reaches 48.0% winrate versus 38.0% for DPO.
- chiPO is robust across two orders of magnitude of beta, whereas DPO degrades completely at small beta.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DPO's logarithmic link with χPO's χ²-based link | Pythia-1b alignment · TL;DR (β=0.05, 1 epoch) | winrate (%): 55.8 ± 2.1 -> 56.5± 1.3 | No clear effect |
| Replace DPO's logarithmic link with χPO's χ²-based link | Pythia-1b alignment · TL;DR (β=0.05, 2 epochs) | winrate (%): 50.3 ± 0.8 -> 56.1 ± 0.6 | Helped |
| Replace DPO's logarithmic link with χPO's χ²-based link | Pythia-1b alignment · TL;DR (β=0.05, 4 epochs) | winrate (%): 38.0 ± 0.7 -> 48.0 ± 1.6 | Helped |
| Replace DPO's logarithmic link with χPO's χ²-based link | Pythia-1b alignment · TL;DR (β=0.005, 1 epoch) | winrate (%): 14.7 ± 3.9 -> 50.6 ± 1.6 | Helped |
| Replace DPO's logarithmic link with χPO's χ²-based link | Pythia-1b alignment · TL;DR (β=0.005, 2 epochs) | winrate (%): 3.4 ± 1.5 -> 52.8 ± 2.3 | Helped |
| Replace DPO's logarithmic link with χPO's χ²-based link | Pythia-1b alignment · TL;DR (β=0.005, 4 epochs) | winrate (%): 0.5 ± 0.2 -> 51.6 ± 0.8 | Helped |
| Decrease β from 0.05 to 0.005 | DPO · TL;DR, 1 epoch | winrate (%): 55.8 ± 2.1 -> 14.7 ± 3.9 | Hurt |
| Decrease β from 0.05 to 0.005 | DPO · TL;DR, 2 epochs | winrate (%): 50.3 ± 0.8 -> 3.4 ± 1.5 | Hurt |
| Decrease β from 0.05 to 0.005 | DPO · TL;DR, 4 epochs | winrate (%): 38.0 ± 0.7 -> 0.5 ± 0.2 | Hurt |
Surprising findings
- DPO decreases the probabilities of both preferred and rejected responses in a constructed example under function approximation.
- DPO+SFT fails to achieve single-policy concentrability for non-convex reward classes, even though it was proposed to do so.
Limitations
- Experiments are preliminary and limited to a single task (TL;DR summarization) with one base model.
- Guarantees depend on bounded implicit rewards (V_max) and realizability of the optimal mixed chi-squared policy.
- Impossibility result shows single-policy concentrability is unattainable for general preference models; only weaker local coverage is obtained via iterative variant.
Experimental setup
Datasets: TL;DR summarization preference dataset
Benchmarks: TL;DR summarization
Metrics: winrate vs baseline KL divergence to reference
Baselines: DPO DPO+SFT
Builds on: DPO RLHF with KL-regularization f-divergence DPO generalization DPO+SFT
Closest papers by meaning
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.91)
- KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity (NeurIPS 2025 · similarity 0.91)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.91)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.91)
- Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.90)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.