Research map / Reinforcement learning / Imitation learning
EvIL: Evolution Strategies for Generalisable Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EvIL, which combines a stabilized IRL variant (IRL++ with reward ensembles, policy buffer, random resets) with an evolution-strategies-learned potential-based shaping term to make recovered rewards both retrainable and sample-efficient in source and transferred MuJoCo environments.
Problem
Rewards recovered by deep IRL methods (e.g. GAIL-style primal methods) often do not permit effective retraining from scratch, even in the source environment, and are poorly shaped so retraining is interaction-inefficient; moment-matching IRL losses are invariant to potential-based shaping.
Approach
Make IRL rewards retrainable via reward/policy ensembles, a full trajectory buffer, and random policy resets (IRL++), then learn a potential-based shaping term by directly maximising the area under the retraining learning curve using OpenAI-ES, rather than approximating the expert value function.
Key results
- In the Ant environment, EvIL achieves a score thousands higher than the unshaped IRL baseline, and matches or exceeds expert performance on Hopper and Humanoid.
- Under 5% action trembling and random link-length variations, EvIL consistently outperforms BC and unshaped baselines in transfer retraining.
- Ablation on Ant: removing policy resets drops EvIL's reward correlation from 0.4082 to 0.0951 (1 agent)-- the resets and buffer contribute most of the gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add replay buffer (ablation) | EvIL · Ant, 5 agents | reward correlation: 0.5739 ($\pm$ 0.0192) -> 0.6696 ($\pm$ 0.0176) | Helped |
| Add replay buffer (ablation) | EvIL · Ant, 1 agent | reward correlation: 0.2702 ($\pm$ 0.1204) -> 0.4082 ($\pm$ 0.1166) | Helped |
| Add L2 regularisation (ablation) | EvIL · Ant, 5 agents | reward correlation: 0.6124 ($\pm$ 0.0233) -> 0.6696 ($\pm$ 0.0176) | Helped |
| Add L2 regularisation (ablation) | EvIL · Ant, 1 agent | reward correlation: 0.3942 ($\pm$ 0.0861) -> 0.4082 ($\pm$ 0.1166) | No clear effect |
| Use the standard-size reward network instead of a bigger reward network (ablation) | EvIL · Ant, 5 agents | reward correlation: 0.5058 ($\pm$ 0.0057) -> 0.6696 ($\pm$ 0.0176) | Helped |
| Use the standard-size reward network instead of a bigger reward network (ablation) | EvIL · Ant, 1 agent | reward correlation: 0.2603 ($\pm$ 0.0901) -> 0.4082 ($\pm$ 0.1166) | Helped |
| Add learner policy resets (ablation) | EvIL · Ant, 5 agents | reward correlation: 0.2544 ($\pm$ 0.0170) -> 0.6696 ($\pm$ 0.0176) | Helped |
| Add learner policy resets (ablation) | EvIL · Ant, 1 agent | reward correlation: 0.0951 ($\pm$ 0.1111) -> 0.4082 ($\pm$ 0.1166) | Helped |
| Use 5 agents instead of 1 agent (ablation) | EvIL · Ant | reward correlation: 0.4082 ($\pm$ 0.1166) -> 0.6696 ($\pm$ 0.0176) | Helped |
| Use 5 agents instead of 1 agent (ablation) | EvIL - No buffer · Ant | reward correlation: 0.2702 ($\pm$ 0.1204) -> 0.5739 ($\pm$ 0.0192) | Helped |
Surprising findings
- Expert value function shaping (V_E) is inconsistent: it hinders retraining in Hopper, Walker and Humanoid but helps in Ant.
- Naively retraining on the final recovered IRL reward often yields negative return in Ant and plateaus at suboptimal performance in other environments, even in the source environment.
Limitations
- Evolving the shaping term requires repeated RL retraining in simulation, which is expensive and only justified when a cheap accurate simulator exists.
- Shaping term evolved on the source environment may suffer from sim-to-real gap; transfer evaluated only on modified MuJoCo variants.
- Results on V_E baseline are inconsistent, and training runs had not converged after 300 generations.
Experimental setup
Datasets: MuJoCo Hopper expert demonstrations MuJoCo Walker expert demonstrations MuJoCo Ant expert demonstrations MuJoCo Humanoid expert demonstrations
Benchmarks: MuJoCo Hopper MuJoCo Walker MuJoCo Ant MuJoCo Humanoid
Metrics: ground-truth episode return area under the learning curve (sample efficiency) correlation between recovered and true reward
Baselines: behavioural cloning unshaped IRL IRL + resets expert value function shaping (V_E) GAIL-style IRL
Builds on: GAIL AIRL PPO OpenAI-ES potential-based reward shaping
Closest papers by meaning
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.90)
- Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.90)
- Environment Design for Inverse Reinforcement Learning (ICML 2024 · similarity 0.89)
- Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment (NeurIPS 2024 · similarity 0.89)
- Efficient Imitation under Misspecification (ICLR 2025 · similarity 0.89)
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.