Research map / Reinforcement learning / Imitation learning

Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration

ICLR 2025 · Reinforcement learning › Imitation learning · approach: Behavior cloning theory and models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ILDE, an imitation learning method combining a GAIL/VAIL-style discriminator reward, a GIRIL curiosity reward from a pretrained VAE dynamics model, and a state-entropy exploration bonus, trained with PPO, to reach beyond-expert performance from very limited demonstrations.

Problem

Imitation learning from very few demonstrations in high-dimensional environments (Atari, MuJoCo) where standard methods fail to match the expert and struggle to reach beyond-expert performance with sample efficiency.

Approach

Integrates a discriminator-based imitation reward with two exploration sources: an optimistic exploration bonus (state entropy via k-NN in a feature space) and curiosity-driven intrinsic reward (pretrained VAE model reconstruction error on demonstration transitions), optimized by PPO; a theoretical ILDE-NPG variant with a sublinear regret bound is also given.

Key results

• On six Atari games with 10% of one-life demonstrations, ILDE outperforms the expert in all 6 games and averages 5.33x expert return versus 4.87x for GIRIL and 0.37x for VAIL.
• ILDE achieves the best average return on the four MuJoCo tasks, e.g. Hopper 1,878±690 vs GIRIL 1,447±273 and VAIL 1,264±783.
• The paper claims a regret bound of Õ(H^{8/3}(dim·log N)^{1/3} T^{2/3} + ε_E T) for the theoretical ILDE-NPG variant.

Measured effects reported in the paper

Surprising findings

• Pure VAIL imitation reward performs worse than the expert on most Atari games (average 0.37x expert), showing pure imitation fails with very limited demonstrations.
• ILDE's advantage on MuJoCo is smaller than on Atari, which the authors attribute to the curiosity reward being better aligned with novelty-seeking tasks.

Limitations

• The method assumes the target task aligns with novelty-seeking or curiosity, which the authors note as a key limitation.
• The theoretical analysis does not directly apply to the practical implementation, as stated in the conclusion.
• Gains on MuJoCo are less significant than on Atari.

Experimental setup

Datasets: Atari one-life demonstrations generated by PPO expert MuJoCo demonstrations from pytorch-a2c-ppo-acktr-gail

Benchmarks: Atari (BeamRider, DemonAttack, BattleZone, Qbert, Krull, StarGunner) MuJoCo (Reacher, Hopper, Walker2d, HumanoidStandup)

Metrics: average return sample efficiency number of games outperforming expert

Baselines: VAIL GIRIL IQ-Learn HyPE

Builds on: GAIL VAIL GIRIL state entropy PPO

Closest papers by meaning

• IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic (ICML 2025 · similarity 0.89)
• Imitation Learning from Observation with Automatic Discount Scheduling (ICLR 2024 · similarity 0.89)
• Limited Preference Aided Imitation Learning from Imperfect Demonstrations (ICML 2024 · similarity 0.89)
• Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.89)
• Efficient Imitation under Misspecification (ICLR 2025 · similarity 0.89)
• Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations (NeurIPS 2025 · similarity 0.88)