Research map / Reinforcement learning / Imitation learning
Hybrid Inverse Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes hybrid inverse RL (HyPE model-free, HyPER model-based) that trains the IRL policy-search inner loop on a mix of learner and expert data, reducing exploration without requiring resets to arbitrary states.
Problem
Inverse reinforcement learning requires repeatedly solving an expensive RL problem in its inner loop, causing high interaction complexity, especially exploring policies far from the expert's.
Approach
Reduce IRL to expert-competitive RL (ERROr oracle) so that any hybrid RL algorithm mixing expert and on-policy data can serve as the policy search inner loop; instantiates this with HyQ (model-free, HyPE) and LAMPS-style model-based training with expert resets inside a learned model (HyPER), without requiring generative resets to the real environment.
Key results
- On MuJoCo, HyPE and HyPER converge to expert performance faster than BC, IQLearn, MM, BC-Reg and FILTER, with the gap growing on harder environments.
- On antmaze-large, all baseline interactive IRL methods achieve 0 reward, while HyPE + Resets surpasses BC and HyPER matches it with roughly 1/10th the environment interaction.
- HyPER reaches the highest reported IRL performance on antmaze-large without resetting to expert states in the real environment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use hybrid expert-and-learner data during inverse RL policy updates | HyPE and HyPER · MuJoCo locomotion | environment interactions to reach expert performance | Helped |
| use hybrid expert-and-learner data during inverse RL policy updates | HyPE and HyPER · MuJoCo locomotion | reward | Helped |
| use HyPE instead of prior interactive methods | D4RL antmaze-large · antmaze-large | reward | Helped |
| use HyPER with expert resets in the learned model rather than real-environment resets | D4RL antmaze-large · antmaze-large | reward | Helped |
| use HyPER instead of HyPE + Resets | D4RL antmaze-large · antmaze-large | environment interactions to match BC performance: roughly 1/10th | Helped |
| reduce expert demonstrations from 64 to 5 trajectories (ablation) | HyPE · MuJoCo low-data regime | convergence speed | Helped |
| reduce expert demonstrations from 64 to 5 trajectories (ablation) | HyPER · MuJoCo low-data regime | performance | Hurt |
| replace MM with classical GAIL (ablation) | Humanoid · Humanoid | performance | Hurt |
| use interactive baseline methods without expert-action leverage | interactive baselines · D4RL antmaze-large | reward: 0 | Hurt |
Surprising findings
- HyPER performs poorly in the low-data regime (5 demonstrations), attributed to difficulty fitting an accurate model.
- A GAIL implementation performs far worse than the MM baseline on Humanoid, indicating MM is a strong IRL baseline.
Limitations
- HyPER struggles with few expert demonstrations because model learning requires sufficient data.
- Guarantees assume realizability of the expert policy; robustness to misspecification is left as a conjecture.
- Model-free HyPE alone fails to match BC on antmaze without real-environment resets.
Experimental setup
Datasets: MuJoCo expert demonstrations (Ant, Hopper, Humanoid, Walker2d) D4RL antmaze-large
Benchmarks: MuJoCo locomotion (Ant, Hopper, Humanoid, Walker2d) D4RL antmaze-large
Metrics: episode reward sample efficiency (environment steps)
Baselines: BC IQLearn MM BC-Reg FILTER GAIL
Builds on: GAIL HyQ LAMPS FILTER SAC
Closest papers by meaning
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.93)
- Inverse Reinforcement Learning by Estimating Expertise of Demonstrators (AAAI 2025 · similarity 0.91)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.91)
- Simplifying Constraint Inference with Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data (AAAI 2025 · similarity 0.91)
- Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.