Research map / Reinforcement learning / Imitation learning
Simplifying Constraint Inference with Inverse Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that inverse constrained RL reduces to plain maximum-entropy IRL for convex-cone constraint classes, and proposes simple IRL modifications (L2 regularization, separate critics, last-layer policy reset) that outperform prior Lagrangian-based ICRL on MuJoCo benchmarks.
Problem
Constraint inference from expert trajectories in continuous control is hard because prior inverse constrained RL methods rely on complex tri-level optimization that yields unstable, suboptimal results.
Approach
Under convex-cone constraint classes, the tri-level ICRL objective equals a bi-level IRL objective, so plain MaxEnt-style IRL with a positive-valued constraint network (L2 regularization on expert costs), separate reward/constraint critics, and last-layer resets can infer constraints without a Lagrangian inner loop.
Key results
- IRL-Plus (IRL with L2 regularization, separate critics, policy reset) outperforms the Lagrangian-based ICRL baseline MECL with statistical significance across all four feasible-reward aggregate metrics in expert-normalized MuJoCo results.
- On Half Cheetah with 20% sub-optimal expert trajectories, IRL-Plus achieves higher feasible rewards than MECL with statistical significance; at 50% and 80% the gap is not significant, and GACL achieves no feasible rewards.
- Basic IRL (IRL-Base) performs comparably to GACL and favorably versus MECL in violation rate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IRL-Base instead of MECL | Constraint inference on five MuJoCo environments · Five MuJoCo environments | feasible rewards | Helped |
| Use IRL-Base instead of MECL | Constraint inference on five MuJoCo environments · Five MuJoCo environments | violation rate | Helped |
| Use IRL-Base instead of GACL | Constraint inference on five MuJoCo environments · Five MuJoCo environments | feasible rewards | No clear effect |
| Use IRL-Base instead of GACL | Constraint inference on five MuJoCo environments · Five MuJoCo environments | violation rate | Helped |
| Add L2 regularization, separate critics, and policy resets (IRL-Plus) | IRL-Base constraint inference · Five MuJoCo environments | feasible rewards | Helped |
| Add L2 regularization, separate critics, and policy resets (IRL-Plus) | IRL-Base constraint inference · Five MuJoCo environments | violation rate | Mixed |
| Add L2 regularization instead of clipping with reward normalization (ablation) | IRL-Base constraint inference · Five MuJoCo environments | feasible rewards | Helped |
| Add L2 regularization instead of clipping with reward normalization (ablation) | IRL-Base constraint inference · Five MuJoCo environments | violation rate | No clear effect |
| Add separate critics without policy resets (ablation) | IRL-L2 · Five MuJoCo environments | feasible rewards | Hurt |
| Add policy resets without separate critics (ablation) | IRL-L2 · Five MuJoCo environments | feasible rewards | No clear effect |
| Add policy resets without separate critics (ablation) | IRL-L2 · Five MuJoCo environments | violation rate | Hurt |
| Add separate critics and policy resets together (ablation) | IRL-L2 · Five MuJoCo environments | feasible rewards | Helped |
Surprising findings
- Separate critics alone and policy resets alone do not improve over IRL+L2, but their combination yields a gain.
- The authors could not reproduce the SAC-based ICRL results of Liu et al. 2023a across six learning-rate settings.
Limitations
- High variance in IRL solutions across seeds and environments, especially Inverted Pendulum.
- Differences from prior ICRL implementation (SAC vs PPO) mean the paper cannot conclude IRL is strictly better in all cases.
- Equivalence theorem relies on convex-cone constraint classes; approximate deep parameterizations fall outside the regret-bound assumptions.
Experimental setup
Datasets: Liu et al. 2023a constraint inference MuJoCo benchmark (Ant, Half Cheetah, Walker2D, Swimmer, Inverted Pendulum)
Benchmarks: Ant Half Cheetah Walker2D Swimmer Inverted Pendulum
Metrics: feasible rewards violation rate IQM median mean optimality gap
Baselines: MECL GACL IRL-Base
Builds on: Malik et al. 2021 maximum entropy ICRL Kim et al. 2023 ICRL game formulation Zeng et al. 2022 IRL implementation Liu et al. 2023a ICRL benchmark Lyle et al. 2023 plasticity
Closest papers by meaning
- Understanding Constraint Inference in Safety-Critical Inverse Reinforcement Learning (ICLR 2025 · similarity 0.92)
- Learning Shared Safety Constraints from Multi-task Demonstrations (NeurIPS 2023 · similarity 0.91)
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.91)
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.91)
- Robust Inverse Constrained Reinforcement Learning under Model Misspecification (ICML 2024 · similarity 0.91)
- Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.