Research map / Large language models / Alignment and preferences
KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives the exact closed-form solution of reverse-KL-regularized RLHF with multiple reference models (a generalized geometric-mean reference), and an implicit solution for forward-KL, with sample-complexity bounds and small GSM8K/UltraFeedback experiments using Qwen2.5 models.
Problem
RLHF regularized by KL to a single reference model ignores other available pretrained models; with multiple references under reverse KL, no exact solution or sample-complexity guarantees existed (prior work only gave a lower-bound approximation).
Approach
Exact closed-form optimum for weighted multi-reference reverse-KL RLHF, given by a normalized weighted geometric mean (generalized escort) of the reference policies tilted by exp(gamma r); the paper also gives an implicit solution for forward-KL with an arithmetic-mean reference, and derives sample-complexity bounds for both.
Key results
- On the 7B model with alpha=0.5 and Qwen2.5-14B-Instruct as second reference, DPO with both references reaches a 66.1% win rate vs 56.4% for single-reference SFT DPO and 59.8% for single-reference 14B DPO.
- The paper proves sub-optimality gap sample complexity O(1/n) and optimality gap O(1/sqrt(n)) for reverse-KL multi-reference RLHF, and O(1/sqrt(n)) for both gaps under forward KL.
- The exact reverse-KL solution is reported to outperform the MRPO approximation and single-reference objective in both online GRPO on GSM8K and offline RLHF on UltraFeedback (95% CI reported).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use both the SFT model and Qwen2.5-14B Instruct as references instead of only Qwen2.5-14B Instruct | DPO on Qwen2.5-7B · UltraFeedback | win rate against preferred answer: 59.8% -> 66.1% | Helped |
Limitations
- Relies on bounded reward assumption (rewards in [0, R_max]) and finite reward class, which may not hold for practical reward models.
- Bounds scale exponentially in R_max and linearly in the local coverage constant.
- Experiments are small-scale (0.5B, 7B) and the paper does not study unbounded rewards.
Experimental setup
Datasets: GSM8K UltraFeedback
Benchmarks: GSM8K UltraFeedback
Metrics: pass@1 normalized reward win rate
Baselines: MRPO approximation single reference DPO DPO with forward KL (arithmetic mean) Base model (Qwen 2.5 7B) SFT model
Builds on: DPO GRPO MRPO
Closest papers by meaning
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.91)
- RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.91)
- Understanding the Effects of RLHF on LLM Generalisation and Diversity (ICLR 2024 · similarity 0.91)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.91)
- Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization (ICLR 2025 · similarity 0.91)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.