Research map / Reinforcement learning / Imitation learning

Provably Efficient Exploration in Inverse Constrained Reinforcement Learning

ICML 2025 · Reinforcement learning › Imitation learning · approach: Theory of inverse RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a strategic exploration framework for inverse constrained RL with feasible-cost-set analysis and two sample-efficient exploration algorithms (BEAR and PCSE) with sample complexity bounds, evaluated on gridworlds and Point Maze.

Problem

Inverse constrained RL requires samples from interactive environments, but the efficacy and efficiency of sampling/exploration strategies for recovering feasible constraints from expert demonstrations are unclear, and no theoretical sample-complexity guarantees exist.

Approach

Defines a feasible cost set for ICRL and bounds cost estimation error by errors in estimated transitions and expert policy; then derives two exploration algorithms: BEAR (minimize aggregate bounded cost-estimation error via an RL solver with reward equal to the error bound) and PCSE (restrict exploration to a candidate set of plausibly optimal policies, solved via a Lagrangian dual with two-timesc

Key results

• PCSE converges fastest among six exploration strategies across four Gridworld environments, per training curves of reward, cost and WGIoU.
• In Gridworld-2 and Gridworld-4, WGIoU converges to a value below 1 because some ground-truth constraints lie in non-critical regions where ICRL infers only the minimal set.
• PCSE scales to a continuous Point Maze environment, where it infers constraints shown as blue dots near the maze center.

Measured effects reported in the paper

Surprising findings

• WGIoU converges below 1 in some gridworlds because ground-truth constraints in non-critical regions are not identifiable from expert behavior, which is expected by the minimal-set definition.

Limitations

• Sample complexity depends directly on state-space size, so the method struggles to scale to large or continuous state spaces.
• Assumes the expert policy is optimal w.r.t. rewards among safe policies and deterministic for soft constraints; assumes a finite discrete state-action space.
• Assumes access to expert policy demonstrations of the online kind; analysis limited to infinite-horizon discounted setting.

Experimental setup

Metrics: discounted cumulative rewards discounted cumulative costs WGIoU

Baselines: random exploration epsilon-greedy exploration maximum-entropy exploration upper confidence bound exploration

Builds on: inverse constrained reinforcement learning maximum entropy inverse RL upper confidence bound exploration

Closest papers by meaning

• Robust Inverse Constrained Reinforcement Learning under Model Misspecification (ICML 2024 · similarity 0.91)
• Recent Advancements in Inverse Reinforcement Learning (AAAI 2024 · similarity 0.90)
• Simplifying Constraint Inference with Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
• Confidence Aware Inverse Constrained Reinforcement Learning (ICML 2024 · similarity 0.89)
• Learning Constraints from Offline Demonstrations via Superior Distribution Correction Estimation (ICML 2024 · similarity 0.89)
• How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach (NeurIPS 2024 · similarity 0.88)