Research map / Large language models / Alignment and preferences

On a Connection Between Imitation Learning and RLHF

ICLR 2025 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that RLHF/DPO can be viewed as imitation learning on the chosen-response distribution with reverse KL, and proposes DIL, a density-ratio-based offline alignment objective that outperforms DPO, SimPO and other baselines on UltraFeedback, TL;DR and Anthropic-HH.

Problem

Preference alignment of LLMs via offline methods such as DPO relies on the Bradley-Terry assumption and reward maximization, which may overfit and degrade reasoning; the paper asks whether RLHF can be understood and improved from an imitation learning perspective.

Approach

Reinterprets RLHF as imitation learning of the chosen-response distribution (forward KL for an energy-based policy, then reverse-KL distillation), and directly minimizes the reverse KL to the chosen distribution by estimating the density ratio pi_chosen/pi_ref with a Bregman-divergence (e.g., LSIF) loss, yielding a hyperparameter-free objective that drives the policy's rejected likelihood down whi

Key results

• On Mistral-7B-Base, DIL w/ LSIF reaches 43.59 on BBH and 2.95 on MATH, outperforming SimPO and DPO on most Open LLM Leaderboard tasks.
• On Llama3-8B, DIL w/ LSIF scores 48.98 on GSM8K versus 31.54 for SimPO and 38.67 for DPO.
• On Pythia-2.8b, DIL w/ LSIF achieves a 67.86 average GPT-4 win rate on TL;DR and 69.17 on Anthropic-HH, above the best baseline CPO (66.01 and 67.86).

Measured effects reported in the paper

Surprising findings

• DPO and SimPO hurt performance on reasoning-heavy tasks such as GSM8K in the reported runs.
• The paper reports that DIL shows the smallest decline in chosen-response likelihood during training, unlike DPO and SimPO whose chosen likelihood drops below zero.

Limitations

• Analysis is limited to the offline setting and does not cover on-policy learning.
• Only three density-ratio losses (LSIF, BCE, UKL) are studied in depth.
• Reported gains on some benchmarks are small, and the density ratio is approximated with rejected responses from the reference distribution.

Experimental setup

Datasets: UltraFeedback Binarized Reddit TL;DR summarization Anthropic-HH

Benchmarks: MMLU-PRO BBH MUSR MATH GSM8K ARC TL;DR summarization win rate Anthropic-HH win rate

Metrics: accuracy win rate

Baselines: DPO f-DPO IPO SLiC CPO SimPO KTO

Builds on: DPO SimPO behavior cloning least-squares importance fitting contrastive predictive coding

Closest papers by meaning

• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.91)
• Clone-Robust AI Alignment (ICML 2025 · similarity 0.91)
• RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.90)
• Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.90)
• Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.90)
• Imitating Language via Scalable Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.90)