Research map / Reinforcement learning / Imitation learning

Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms

ICML 2024 · Reinforcement learning › Imitation learning · approach: Theory of inverse RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces a new feasible-reward-set notion for offline IRL, sub/super feasible sets learnable from expert and behavioral datasets, and two tabular algorithms (IRL-LO, PIRL-LO) with PAC sample complexity bounds, with PIRL-LO adding inclusion monotonicity via pessimism.

Problem

Inverse RL from offline datasets of expert and behavioral trajectories is ill-posed and, under partial coverage, the classical feasible reward set is not learnable; the paper seeks learnable solution concepts and efficient algorithms.

Approach

Redefine the feasible set to require expert optimality only on states reachable by the expert, define learnable sub- and super-feasible sets via the transition-model equivalence class, and use a pessimistic confidence-set construction to guarantee inclusion monotonicity of the estimated sets, with new semimetrics that handle unbounded rewards.

Key results

• IRL-LO is (epsilon,delta)-PAC for d-IRL with tau^E constant in epsilon and tau^b scaling as H^3 Z ln(1/delta)/epsilon^2 times (ln(1/delta)+S_max), with O(HS^2A) time.
• PIRL-LO is inclusion monotonic with an extra H^6/(rho_min epsilon^2) sample term, at O(HS^2A log S) time.
• Proves that with expert-only data no reward in the sub-feasible set can assign a non-expert action a higher reward than the expert action.

Surprising findings

• With only expert data, learnable sub-feasible rewards must weakly favor the expert action in every expert-visited state, making IRL close to behavioral cloning.

Limitations

• Results restricted to tabular finite MDPs
• Lower bounds and minimax optimality not established
• Assumes expert's covering assumption for simplicity

Experimental setup

Datasets: expert trajectories D^E behavioral trajectories D^b (tabular MDP samples) highway driving data (Appendix sanity check)

Metrics: sample complexity (PAC bounds) Hausdorff distance between reward sets inclusion monotonicity

Baselines: Metelli et al. 2023 offline reward mapping approach online generative model IRL AceIRL

Builds on: Metelli et al. 2021 feasible reward set Lazzati et al. online IRL feasible set PEVI pessimistic value iteration extended value iteration

Closest papers by meaning

• Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.95)
• Recent Advancements in Inverse Reinforcement Learning (AAAI 2024 · similarity 0.95)
• How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach (NeurIPS 2024 · similarity 0.92)
• Sub-optimal Experts mitigate Ambiguity in Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
• Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.90)
• Inverse Reinforcement Learning by Estimating Expertise of Demonstrators (AAAI 2025 · similarity 0.90)