Research map / Reinforcement learning / Imitation learning
Sub-optimal Experts mitigate Ambiguity in Inverse Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends IRL feasible-reward-set recovery to multiple sub-optimal experts with known sub-optimality bounds, showing they shrink the feasible set and deriving sample complexity bounds with a minimax-optimal uniform sampling algorithm when experts are near-optimal.
Problem
Inverse reinforcement learning is ill-posed: many reward functions are compatible with an optimal expert. The paper asks whether demonstrations from additional sub-optimal experts (with known sub-optimality bounds) can shrink the feasible reward set and what the statistical complexity of estimating
Approach
Formulates IRL with multiple sub-optimal experts (IRL-SE) where each expert's value gap to the optimal expert is bounded by a known xi_i; gives an explicit characterization of the feasible reward set via linear constraints on the advantage variable zeta using expert occupancy measures, and a uniform sampling algorithm matching a lower bound when pi_min^{-1} xi_i <= 1.
Key results
- Theorem 3.6 gives an if-and-only-if characterization of the feasible reward set with sub-optimal experts via linear constraints d^{pi_Ei} pi_Ei B_{pi_E1} zeta <= xi_i.
- Lower bound on sample complexity Omega(SA log(1/delta)/(eps^2 (1-gamma)^2) + S) and a further term scaling with pi_min^{-1}, showing sub-optimal experts do not reduce statistical complexity.
- US-IRL-SE is minimax optimal (matches the lower bound) when pi_min^{-1} max_i xi_i <= 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multiple behaviorally different sub-optimal experts with sufficiently small sub-optimality bounds | IRL-SE feasible reward set · IRL-SE | feasible-set size | Helped |
| Add sub-optimal experts whose policies match the optimal expert, or whose bounds satisfy ξi ≥ (1 − γ)−1 | IRL-SE feasible reward set · IRL-SE | feasible-set size | No clear effect |
| Estimate the feasible reward set with sub-optimal experts rather than only the optimal expert | IRL-SE identification · Generative-model identification | sample complexity | Hurt |
| Use US-IRL-SE uniform sampling when πmin−1ξi ≤ 1 for all sub-optimal experts | IRL-SE identification · Generative-model identification | sample complexity | Helped |
Surprising findings
- Sub-optimal experts do not reduce statistical complexity of IRL; they add a lower-bound term scaling with the inverse of the minimum action probability of sub-optimal experts.
Limitations
- Requires a generative model and known sub-optimality bounds xi_i; offline setting left open.
- Upper and lower bounds do not match when experts are not sufficiently close to optimal.
- Results are tabular; no practical algorithm for large state-action spaces.
Experimental setup
Metrics: Hausdorff distance sample complexity feasible set volume
Baselines: single-expert IRL feasible set (RB) uniform sampling single-expert IRL
Builds on: feasible reward set characterization (Metelli et al. 2021) PAC framework for IRL (Lazzati et al. 2023)
Closest papers by meaning
- Recent Advancements in Inverse Reinforcement Learning (AAAI 2024 · similarity 0.96)
- Towards Theoretical Understanding of Inverse Reinforcement Learning (ICML 2023 · similarity 0.92)
- Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms (ICML 2024 · similarity 0.92)
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.91)
- Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Inverse Reinforcement Learning by Estimating Expertise of Demonstrators (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.