Research map / Reinforcement learning / Imitation learning

When Does Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning

NeurIPS 2026 · Reinforcement learning › Imitation learning · approach: Behavior cloning theory and models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OVI, an interactive on-policy value-based imitation learning algorithm that needs only expert value realizability, proves an offline lower bound showing interaction is necessary, and validates it on Gymnasium tasks with small learner networks.

Problem

Imitation learning with a learner that cannot represent the expert's policy (e.g., in distillation) suffers from compounding errors and plateaus; the paper asks when interaction and value-based learning help and what representational assumptions suffice.

Approach

Interaction relaxes representational demands: an on-policy algorithm (OVI) that layer-wise solves a min-max game between a policy player and a Q player only needs the expert's value function to be realizable (Q^expert-realizability), not the expert's policy; offline value-based IL provably needs complexity scaling with the state space or policy class.

Key results

• OVI outperforms BC, DAgger, and SPOIL across learner widths {2,...,64}, with the largest gains for small learner networks; SPOIL degrades faster than OVI on CartPole-v1 and Acrobot-v1.
• Theory: offline IL under Q^expert-realizability requires Omega(min{|X|, log|F|}/eps) expert trajectories (H=2 construction), while OVI achieves sample complexity polynomial in log|Q|, H, 1/eps.
• Reward realizability alone is insufficient for interactive IL: lower bound Omega(X/eps) rounds of interaction for H=2.

Surprising findings

• DAgger is surprisingly suboptimal in a synthetic Q*-realizable MDP, with its TV distance to the expert staying roughly constant while OVI converges faster in return.
• Value-induced offline policy classes are unidentifiable: even with infinite data they can fail to compete with the expert (constant error 1/4).

Limitations

• Nonconvex classes get suboptimal eps^-4 rate; the eps^-2 rate requires the inefficient Q-OVI variant.
• OVI learns nonstationary policies with computation and memory scaling linearly with horizon; experiments use a stationary approximation.
• Language-model distillation empirical validation is left open.

Experimental setup

Datasets: Gymnasium CartPole-v1 Acrobot-v1 other Gymnasium environments (four total)

Benchmarks: CartPole-v1 Acrobot-v1 Gymnasium environments (four total) synthetic Q-realizable MDP

Metrics: expert return suboptimality J^expert - J^pi trajectory-level TV distance

Baselines: BC DAgger SPOIL

Builds on: On-Q moment matching (DAeQuIL) DAgger SPOIL Forward algorithm performance difference lemma

Closest papers by meaning

• Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback (NeurIPS 2026 · similarity 0.90)
• RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.89)
• Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.89)
• Efficient Imitation under Misspecification (ICLR 2025 · similarity 0.89)
• Imitation Learning from Observation with Automatic Discount Scheduling (ICLR 2024 · similarity 0.89)
• Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.89)