Research map / Reinforcement learning / Imitation learning

Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning

NeurIPS 2024 · Reinforcement learning › Imitation learning · approach: Behavior cloning theory and models · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that behavior cloning with log loss (LogLossBC) achieves horizon-independent or linear-horizon sample complexity in offline imitation learning under normalized rewards and parameter sharing, closing much of the offline-online gap.

Problem

Whether offline imitation learning via behavior cloning necessarily incurs quadratic horizon dependence, and whether online imitation learning is fundamentally more sample-efficient than offline IL for general (including neural) policy classes.

Approach

Analyzes behavior cloning with the log loss (maximum likelihood over trajectory distributions) and bounds rollout regret by squared Hellinger distance at the trajectory level, yielding horizon-independent bounds for sparse rewards and linear-in-H bounds for dense rewards under parameter sharing; for stochastic experts, a variance-dependent bound with a recoverability-based refinement.

Key results

• Theory: LogLossBC gives J(pi*)-J(pihat) <= 8R log(2|Pi|/delta)/n, removing the H factor from indicator-loss BC; under dense rewards with parameter sharing this is linear in H.
• Lower bound: without parameter sharing, online IL (including log-loss DAgger) cannot beat offline LogLossBC by more than constant factors in the worst case, with Omega(H/n) lower bound for deterministic experts.
• Experiments on Walker2d and Beamrider show regret roughly independent of horizon H for LogLossBC; Dyck regret increases with H.

Measured effects reported in the paper

Surprising findings

• Online IL cannot improve over offline LogLossBC for policy classes with parameter sharing, contrary to common intuition.
• For stochastic experts, quadratic dependence on horizon is necessary under dense rewards, unlike the deterministic case.

Limitations

• Results are for the realizable setting; misspecification is treated only partially and left largely for future work.
• Stochastic-expert dense-reward bounds retain a slow 1/sqrt(n) term unless recoverability holds; the recoverability-based result relies on a signed recoverability constant.
• Experiments are small-scale and simplified, with language-generation results described as more nuanced.

Experimental setup

Datasets: Walker2d-v4 expert rollouts BeamRiderNoFrameskip-v4 expert rollouts Car navigation synthetic rollouts Dyck_3 synthetic sequences

Benchmarks: Walker2d-v4 BeamRiderNoFrameskip-v4 Car (custom) Dyck_3

Metrics: expected regret vs expert rollout reward

Baselines: indicator-loss behavior cloning DAgger Forward Smile Aggrevate

Builds on: DAgger Aggrevate Hellinger distance change-of-measure Freedman-type concentration

Closest papers by meaning

• Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback (NeurIPS 2026 · similarity 0.92)
• Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift (NeurIPS 2026 · similarity 0.92)
• Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.92)
• How to Leverage Diverse Demonstrations in Offline Imitation Learning (ICML 2024 · similarity 0.91)
• Mitigating Covariate Shift in Behavioral Cloning via Robust Stationary Distribution Correction (NeurIPS 2024 · similarity 0.91)
• Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.90)