Research map / Reinforcement learning / Imitation learning
The MAGICAL Benchmark for Robust Imitation
Sam Toyer, Rohin Shah, Andrew Critch, Stuart J. Russell
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Test on Layout variant instead of Demo variant (object positions changed) | BC · MAGICAL | mean test score: 0.64±0.29 -> 0.14±0.16 | Hurt |
| Test on Layout variant instead of Demo variant | GAIL · MAGICAL | mean test score: 0.72±0.35 -> 0.22±0.23 | Hurt |
| Test on Colour variant instead of Demo variant | GAIL · MAGICAL | mean test score: 0.72±0.35 -> 0.27±0.24 | Hurt |
| Remove all augmentations (ablation) | BC · MAGICAL Jitter variant | mean test score: 0.56±0.27 -> 0.37±0.30 | Hurt |
| Remove all augmentations (ablation) | BC · MAGICAL Demo variant | mean test score: 0.64±0.29 -> 0.55±0.37 | Hurt |
| Remove all discriminator augmentations (ablation) | GAIL · MAGICAL Demo variant | mean test score: 0.72±0.35 -> 0.44±0.42 | Hurt |
| Use allocentric instead of egocentric perspective (ablation) | BC · MAGICAL Layout variant | mean test score: 0.14±0.16 -> 0.04±0.04 | Hurt |
| Use allocentric instead of egocentric perspective (ablation) | GAIL · MAGICAL Colour variant | mean test score: 0.27±0.24 -> 0.39±0.36 | Improved |
| Use allocentric instead of egocentric perspective (ablation) | BC · MAGICAL Colour variant | mean test score: 0.39±0.30 -> 0.42±0.32 | Improved |
| Switch GAIL discriminator loss to WGAIL-GP (ablation) | GAIL · MAGICAL Demo variant | mean test score: 0.72±0.35 -> 0.42±0.38 | Hurt |
| Switch GAIL discriminator loss to WGAIL-GP (ablation) | GAIL · MAGICAL Colour variant | mean test score: 0.27±0.24 -> 0.10±0.11 | Hurt |
| Train multi-task instead of single-task (ablation) | BC · MAGICAL Demo variant | mean test score: 0.64±0.29 -> 0.59±0.33 | No clear effect |
Closest papers by meaning
- Imitation Learning from Imperfect Demonstration (ICML 2019 · similarity 0.90)
- Quantifying Generalisation in Imitation Learning (NeurIPS 2025 · similarity 0.90)
- DualCOIL: Offline Imitation Learning from Contrasting Demonstrations (ICML 2026 · similarity 0.90)
- Imitation Learning by Estimating Expertise of Demonstrators (ICML 2022 · similarity 0.89)
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.88)
- How to Leverage Diverse Demonstrations in Offline Imitation Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
