Research map / Reinforcement learning / Imitation learning
Inverse Reinforcement Learning by Estimating Expertise of Demonstrators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IRLEED, an IRL framework that jointly estimates the true reward and each demonstrator's reward bias (accuracy) and action precision via a Boltzmann suboptimality model, improving over max-entropy IRL, IQ-Learn and ILEED on heterogeneous suboptimal demonstrations.
Problem
Imitation learning from heterogeneous, suboptimal demonstrations whose quality varies by demonstrator, without prior knowledge of demonstrator expertise; standard IRL averages their behavior and inherits suboptimality.
Approach
Model each demonstrator as a Boltzmann-rational agent with a biased reward r+epsilon_i and precision beta_i, and jointly maximize likelihood over the shared true reward and per-demonstrator accuracy and precision parameters within a maximum-entropy IRL framework; generalizes IRL and ILEED and is extended to generalized IRL with neural networks and IQ-Learn.
Key results
- On average IRLEED gave a 30.3% improvement over IRL in Gridworld.
- In Atari with combined dataset AB, IRLEED shows a 17% improvement over IQ and a 41% improvement over ILEED.
- In Lunar Lander low-precision low-accuracy setting, IRLEED reaches 1.34 relative return vs 0.81 for IQ and 0.84 for ILEED.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add demonstrator-specific accuracy deviations and learnable precision to IQ (ablation) | IQ · Cartpole and Lunar Lander, three suboptimal dataset settings | return relative to best demonstrator | Helped |
| Add demonstrator-specific accuracy deviations and learnable precision to IQ (ablation) | IQ · Space Invaders and Qbert, combined dataset AB | average return: 17% improvement over IQ | Helped |
| Add demonstrator-specific accuracy deviations and learnable precision to IQ (ablation) | IQ · Mujoco Hopper, combined dataset ABC | mean episode return | Helped |
| Add demonstrator-specific accuracy deviations and learnable precision to maximum entropy IRL | Maximum entropy IRL · Gridworld, heterogeneous suboptimal demonstrations | policy performance: 30.3% improvement over IRL | Helped |
| Add demonstrator-specific accuracy deviations and learnable precision to IQ (ablation) | IQ · Space Invaders and Qbert, simulated dataset A | average return | Mixed |
| Add demonstrator-specific accuracy deviations and learnable precision to IQ (ablation) | IQ · Space Invaders and Qbert, human dataset B | average return | Mixed |
| Use the combined simulated and human dataset AB instead of simulated dataset A alone | IRLEED · Space Invaders and Qbert | average return | Mixed |
| Use the combined simulated and human dataset AB instead of human dataset B alone | IRLEED · Space Invaders and Qbert | average return | Helped |
| Add demonstrator-specific accuracy deviations and learnable precision to IQ (ablation) | IQ · Cartpole and Lunar Lander, clean high-precision/high-accuracy demonstrations | return relative to best demonstrator | No clear effect |
| Remove reward-bias modeling (accuracy deviations) (ablation) | IRLEED · Simulated control tasks, low-accuracy demonstrations | return relative to best demonstrator | Hurt |
Surprising findings
- Standard IRL's recovered policy can perform no better than the average demonstrator, as shown by the feature-matching bound.
Limitations
- Learning multiple demonstrator models adds complexity.
- No theoretical guarantee that IRLEED recovers a better reward than IRL; the objective is nonconvex and the true reward is not unique.
- Reward recovery is only analyzed in Gridworld because it is hard to interpret in complex environments.
Experimental setup
Datasets: Gridworld (custom) CartPole Lunar Lander Atari Space Invaders Qbert human Atari demonstrations MuJoCo Hopper
Benchmarks: Gridworld CartPole Lunar Lander Atari Space Invaders Qbert MuJoCo Hopper
Metrics: average return relative to best demonstrator episode return relative improvement over max-entropy IRL
Baselines: maximum entropy IRL IQ-Learn ILEED mean of demonstrators
Builds on: maximum entropy IRL IQ-Learn ILEED
Closest papers by meaning
- Recent Advancements in Inverse Reinforcement Learning (AAAI 2024 · similarity 0.92)
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.92)
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.91)
- Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment (NeurIPS 2024 · similarity 0.91)
- Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms (ICML 2024 · similarity 0.90)
- Sub-optimal Experts mitigate Ambiguity in Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.