Research map / Reinforcement learning / Imitation learning
Robust Inverse Constrained Reinforcement Learning under Model Misspecification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that inverse constrained RL constraints learned in one environment become unsafe under dynamics mismatch, and proposes AR-ICRL, which learns a robust policy via a two-player Markov game with an opponent policy and then distills a robust constraint.
Problem
Inverse constrained RL infers constraints from expert demonstrations assuming training and deployment dynamics are identical; under dynamics mismatch the inferred constraints fail to keep agents safe.
Approach
Model dynamics misspecification as an adversarial opponent policy mixed into the executed policy, solve a two-player Markov game to get a safety-robust policy, adapt the constraint (feasibility function) to match expert likelihood under soft or hard optimality, then distill the robust policy into a robust constraint.
Key results
- In Blocked Half-Cheetah under large fully random noise, AR-ICRL achieved a 31%±10% constraint violation rate versus 40%±18% for MEICRL.
- In Blocked Ant under large attack noise, AR-ICRL reached 44%±16% violation rate versus 56%±30% for MEICRL and 54%±45% for VICRL.
- In windy Gridworlds, AR-ICRL showed notably lower constraint violation than MEICRL under increasing wind strength.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AR-ICRL instead of MEICRL | Blocked Half-Cheetah · Large-scale fully random noise | Constraint Violation Rate: 40%±18% -> 31%±10% | Helped |
| Use AR-ICRL instead of IRCO | Blocked Half-Cheetah · Large-scale partially random noise | Constraint Violation Rate: 43%±8% -> 34%±2% | Helped |
| Use AR-ICRL instead of IRCO | Blocked Half-Cheetah · Large-scale attack noise | Constraint Violation Rate: 72%±13% -> 63%±8% | Helped |
| Use AR-ICRL instead of BC2L | Blocked Ant · Large-scale fully random noise | Constraint Violation Rate: 58%±9% -> 53%±4% | Helped |
| Use AR-ICRL instead of IRCO | Blocked Ant · Large-scale partially random noise | Constraint Violation Rate: 67%±7% -> 59%±6% | Helped |
| Use AR-ICRL instead of VICRL | Blocked Ant · Large-scale attack noise | Constraint Violation Rate: 54%±45% -> 44%±16% | Helped |
| Use AR-ICRL instead of IRCO | Crippled Walker · Large-scale fully random noise | Constraint Violation Rate: 34%±16% -> 31%±9% | Helped |
| Use AR-ICRL instead of IRCO | Crippled Walker · Large-scale partially random noise | Constraint Violation Rate: 69%±24% -> 65%±12% | Helped |
| Use AR-ICRL instead of IRCO | Crippled Walker · Large-scale attack noise | Constraint Violation Rate: 68%±14% -> 53%±37% | Helped |
| Increase opponent strength (lower α) | AR-ICRL · Windy Gridworlds | Constraint Violation Rate | Helped |
Surprising findings
- BC2L and VICRL attain higher rewards in Blocked Ant but cannot guarantee safety, showing reward does not track constraint satisfaction.
Limitations
- Requires online environment interaction, which may be costly or hazardous.
- Assumes the training-deployment dynamics discrepancy lies within a bounded uncertainty set.
- Experiments use only four seeds and rely on a fixed alpha=0.95 for continuous tasks.
Experimental setup
Datasets: ICRL benchmark (Liu et al., 2023) expert demonstrations in Gridworld environments
Benchmarks: Gridworld (windy) Blocked Half-Cheetah Blocked Ant Crippled Walker
Metrics: constraint violation rate episode cost feasible reward
Baselines: MEICRL BC2L VICRL IRCO
Builds on: MEICRL Viano et al. adversarial robust imitation PPO-Lagrange VICRL
Closest papers by meaning
- Confidence Aware Inverse Constrained Reinforcement Learning (ICML 2024 · similarity 0.92)
- Learning Constraints from Offline Demonstrations via Superior Distribution Correction Estimation (ICML 2024 · similarity 0.92)
- Provably Efficient Exploration in Inverse Constrained Reinforcement Learning (ICML 2025 · similarity 0.91)
- Simplifying Constraint Inference with Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Constrained Reinforcement Learning Under Model Mismatch (ICML 2024 · similarity 0.91)
- Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.