Research map / Reinforcement learning / Imitation learning
Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis of how sensitive IRL is to misspecification of the behavioural model, giving necessary and sufficient robustness conditions using STARC metrics and showing that common models (optimality, Boltzmann, MCE) are highly sensitive to perturbations and to misspecified discount or transition parameters.
Problem
Inverse reinforcement learning infers rewards from policies using a behavioural model; if that model is misspecified, the inferred reward may be far from the true one. The paper asks how robust IRL is to such misspecification.
Approach
Formalises misspecification robustness via a reward pseudometric (STARC, which is sound and complete for worst-case regret), gives necessary and sufficient conditions for robustness, and proves that optimality, Boltzmann-rationality and MCE models are not robust to small perturbations, discount misspecification, or transition misspecification, though Boltzmann and MCE are robust to temperature and
Key results
- Necessary and sufficient conditions for epsilon-robustness are given in terms of composing the behavioural model with transformations bounded by epsilon in STARC distance.
- Optimality, Boltzmann-rational and MCE models are not epsilon/delta-separating, so no continuous behavioural model is robust to arbitrarily small policy perturbations under STARC distance.
- Any model invariant to potential shaping is not epsilon-robust (epsilon < 0.5) to misspecification of the discount rate for non-trivial transitions; similarly for S'-redistribution and the transition function.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a different discount parameter γ | Optimality, Boltzmann-rationality, or MCE · Non-trivial transition dynamics; STARC distance | IRL robustness threshold ε: any ε < 0.5 | Hurt |
| Use different transition dynamics τ | Optimality, Boltzmann-rationality, or MCE · STARC distance | IRL robustness threshold ε: any ε < 0.5 | Hurt |
| Perturb the observed policy while using a continuous behavioural model | Boltzmann-rationality or MCE · Reward space containing all rewards with a fixed L2 norm; STARC distance | ε/δ-separation: any ε < 1 or δ > 0 | Hurt |
| Change the Boltzmann temperature β | Boltzmann-rationality · STARC distance | IRL robustness threshold ε: any ε ≥ 0 | No clear effect |
| Change the MCE weight α | MCE · STARC distance | IRL robustness threshold ε: any ε ≥ 0 | No clear effect |
Surprising findings
- Very mild misspecification of the discount rate or transition function can yield very large reward errors, even when the parameters are arbitrarily close.
- EPIC is sensitive to S'-redistribution, so it is unsuitable for analysing these behavioural models.
Limitations
- Results are worst-case and existential; restricting the reward set may circumvent negative results.
- Analysis relies on STARC metrics, which are strong; weaker guarantees may be possible with other metrics.
- Does not cover hyperbolic discounting or other behavioural models, left to future work.
Experimental setup
Baselines: EPIC DARD
Builds on: STARC metric EPIC maximal causal entropy IRL
Closest papers by meaning
- Partial Identifiability in Inverse Reinforcement Learning for Agents with Non-Exponential Discounting (AAAI 2025 · similarity 0.91)
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.90)
- Recent Advancements in Inverse Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Efficient Imitation under Misspecification (ICLR 2025 · similarity 0.89)
- Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms (ICML 2024 · similarity 0.89)
- Learning Utilities from Demonstrations in Markov Decision Processes (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.