Research map / Reinforcement learning / Offline RL

When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Diffusion models for offline decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that offline SFT suffices under realizability, while under non-realizability online imitation learning (on-policy distillation) gains via a misspecification-reward alignment condition and an offline information-theoretic lower bound.

Problem

Explaining when and why online imitation learning (e.g., on-policy distillation) outperforms offline SFT in LLM post-training, beyond the classical error-accumulation account.

Approach

Identifies non-realizability (student class cannot represent expert) as an orthogonal source of online IL advantage, proves an offline coverage-based lower bound even at horizon H=1, and gives a reward-dependent misspecification-reward alignment condition under which online IL is provably effective.

Key results

• Under realizability, SFT with an RL-trained expert matches expert performance and on-policy distillation gives no accuracy or speed gains.
• On-policy distillation from Skywork-7B raises DeepSeek-R1-Distill-1.5B average AIME performance from 25.0% to 32.8%, versus 27.1% from R1-Distill-7B.
• Length-induced TV distance between DeepSeek-R1-0528 and its Qwen3-8B distillate is 29.99%, exceeding a 5.46% performance gap on AIME.

Surprising findings

• Discrepancy-based (TV or Hellinger) bounds can be far too pessimistic: response distribution gaps exceed performance gaps by a wide margin.
• Under realizability, offline SFT matches expert performance, so the SFT-vs-online gap need not reflect an inherent limitation of SFT.

Limitations

• Theory uses a contextual bandit (H=1) abstraction with finite policy classes, not full autoregressive MDP.
• Does not explain why on-policy distillation is more efficient than RLVR.
• Misspecification-reward alignment condition is assumption-based and estimated empirically via a proxy policy class.

Experimental setup

Datasets: Countdown-Tasks-3to4 GSM8K OpenR1 DeepScaleR-40k MATH AIME 2024 AIME 2025 DAPO-Math-17K MMLU

Benchmarks: Countdown GSM8K MATH AIME 2024 AIME 2025 MMLU

Metrics: accuracy average accuracy over 8 samples estimated suboptimality bound

Baselines: SFT GRPO on-policy distillation with realizable expert Hellinger-based RhoEstimatorBC

Builds on: on-policy distillation DAgger GRPO MiniLLM pessimistic policy optimization

Closest papers by meaning

• Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback (NeurIPS 2026 · similarity 0.91)
• Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.90)
• When Does Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning (NeurIPS 2026 · similarity 0.88)
• OLLIE: Imitation Learning from Offline Pretraining to Online Finetuning (ICML 2024 · similarity 0.88)
• Efficient Imitation under Misspecification (ICLR 2025 · similarity 0.88)
• Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.87)