Research map / Reinforcement learning / Imitation learning
Efficient Imitation under Misspecification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes reward-agnostic policy completeness and the GUITAR efficient IRL algorithm, showing local search with expert or offline-data resets avoids compounding errors under misspecification.
Problem
Imitation learning when the learner's policy class cannot represent the expert (misspecification), asking how to perform computationally efficient interactive IRL without compounding errors and where to place local search resets.
Approach
Introduces reward-agnostic approximate policy completeness, under which dual IRL with PSDP resets to expert states (GUITAR) avoids quadratic compounding errors; proposes broadening resets to offline data covering the optimal realizable policy's states.
Key results
- GUITAR, resetting to offline realizable-policy states, outperforms FILTER (expert resets) in misspecified AntMaze variants, with gains growing as misspecification increases.
- GUITAR is the only interactive algorithm that solves the hard exploration problem in the strongly misspecified maze setting.
- In the changing-dynamics MuJoCo experiments, GUITAR shows faster learning than traditional IRL (MM), while FILTER cannot be applied.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reset policy search to states from the optimal realizable policy instead of expert states | GUITAR vs. FILTER · Antmaze-Umaze with unreachable expert states | learning speed | Helped |
| Reset policy search to states from the optimal realizable policy instead of the true starting state | GUITAR vs. MM · Antmaze-Umaze with unreachable expert states | performance | Helped |
| Reset policy search to states from a realizable policy | GUITAR vs. traditional inverse RL methods · MuJoCo continuous control with different expert and learner dynamics | learning speed | Helped |
| Use offline data for direct policy learning rather than for resets | Behavioral cloning · Misspecified continuous control tasks with changing dynamics | performance | Hurt |
Surprising findings
- Behavioral cloning performs consistently poorly across misspecified settings, being sensitive to dynamics shift.
Limitations
- Analysis ignores expert recoverability; cliff-like problems where no policy can recover from one mistake are not captured.
- Theory relies on approximate policy completeness and finite policy/reward classes; the statistically optimal STILE is computationally infeasible.
- Experiments use optimal realizable offline data; sub-optimal data results are deferred to appendix.
Experimental setup
Datasets: expert demonstrations offline data from optimal realizable policy (generated)
Benchmarks: AntMaze-Umaze variants MuJoCo continuous control with altered morphology
Metrics: learning curves / return across seeds standard error
Baselines: BC MM FILTER
Builds on: PSDP MMDP FILTER DAgger Soft Actor Critic
Closest papers by meaning
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.90)
- Imitation Learning from Observation with Automatic Discount Scheduling (ICLR 2024 · similarity 0.89)
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.89)
- Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification (ICLR 2024 · similarity 0.89)
- When Does Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning (NeurIPS 2026 · similarity 0.89)
- Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.