Research map / Reinforcement learning / Imitation learning

Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback

NeurIPS 2026 · Reinforcement learning › Imitation learning · approach: Behavior cloning theory and models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a noisy-expert model for imitation learning and shows offline BC needs exponential horizon dependence while online on-policy distillation variants (NAIL) achieve polynomial or horizon-free guarantees, validated on modular addition and GSM8K.

Problem

Explains why online on-policy distillation outperforms offline SFT/behavior cloning in practice when the expert is noisy, given that offline IL is theoretically horizon-free with clean experts.

Approach

Models the expert as a noisy mixture (1-eta)pi* + eta*nu, shows offline BC suffers exponential horizon dependence under noise, and proposes NAIL, an always-on-policy trajectory-level aggregation/distillation method (greedy student rollouts with augmented-trajectory KL losses) whose guarantees are polynomial in horizon or horizon-free under domination/determinism conditions.

Key results

• On modular addition with eta=0.2 noisy expert, NAIL-F and NAIL-R reach near-perfect accuracy while LogLossBC, OPD-F, and OPD-R fail to learn after over 1M expert interactions.
• On GSM-8K with a temperature-4 noisy teacher (0% accuracy), only NAIL-F and NAIL-R achieve nontrivial accuracy while LogLossBC, OPD-F, OPD-R stay at 0.
• Theory: offline BC sample complexity scales as (1-eta)^{-(H+2)} under noise, while NAIL achieves H/(1-eta) sqrt(log|Pi|/n) and horizon-free rates under deterministic expert with smooth corruption.

Surprising findings

• Offline BC and standard OPD fail completely under high expert noise, even on clean-trained comparisons where they otherwise work.
• Greedy student rollouts combined with augmented-trajectory KL outperform sampled-rollout OPD under noise, contrary to standard OPD practice.

Limitations

• Theory is in the finite policy class setting; extension to infinite classes needs covering arguments.
• Unknown-corruption guarantees only hold for deterministic experts with smooth corruption; a linear-in-horizon gap remains between upper and lower bounds for general experts.
• Experiments are small in scale and the corruption model is idealized (absorbing uniform noise).

Experimental setup

Datasets: TinyGSM GSM-8K modular addition synthetic data (p=7, m=31)

Benchmarks: GSM-8K modular addition

Metrics: exact-match accuracy validation accuracy validation loss

Baselines: LogLossBC OPD-F OPD-R

Builds on: DAgger on-policy distillation behavior cloning horizon-free analysis (Rajaraman et al.) nanoGPT

Closest papers by meaning

• Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.92)
• When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon (NeurIPS 2026 · similarity 0.91)
• Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift (NeurIPS 2026 · similarity 0.91)
• When Does Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning (NeurIPS 2026 · similarity 0.90)
• Offline Actor-Critic Reinforcement Learning Scales to Large Models (ICML 2024 · similarity 0.88)
• Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model (ICLR 2025 · similarity 0.88)