Research map / Reinforcement learning / Imitation learning
Confidence Aware Inverse Constrained Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CA-ICRL, an inverse constrained RL method that learns a Beta-distribution-based constraint whose quantile at a user-chosen confidence level yields a constraint at least as constraining as the ground truth, and tells whether enough expert trajectories were collected.
Problem
Inverse constrained RL estimates constraints from expert demonstrations, but existing methods provide no confidence measure for learned constraints and no way to tell whether the number of demonstrations is sufficient.
Approach
Model the feasibility probability of trajectories with a Beta distribution whose parameters are output by a set-based encoder network over expert trajectories, and use its (1-lambda) quantile as the constraint so it is at least as constraining as the true constraint with confidence lambda; the same procedure indicates when more demonstrations are needed to meet a performance threshold.
Key results
- With 70% confidence, CA-ICRL keeps constraint violation rate below 30% in all MuJoCo environments except Biased Pendulum and obtains more rewards than GACL, BC2L, ICRL and VICRL during training.
- In Blocked Walker with 80% confidence, CA-ICRL's violation rate drops below 20% for 100, 200 and 300 expert trajectories, and its reward is higher than VICRL in all three settings.
- Across confidence levels 30%, 50%, 70%, 90% in Blocked Walker, violation rate decreases and reward decreases as confidence increases, illustrating the trade-off.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CA-ICRL instead of VICRL | ICRL · MuJoCo and HighD environments | rewards | Helped |
| Use CA-ICRL instead of VICRL | ICRL · MuJoCo and HighD environments | constraint violation rate | Helped |
| Increase confidence from 30% to 90% (ablation) | CA-ICRL on Blocked Walker with 150 expert trajectories · Blocked Walker | constraint violation rate | Helped |
| Increase confidence from 30% to 90% (ablation) | CA-ICRL on Blocked Walker with 150 expert trajectories · Blocked Walker | rewards | Hurt |
| Increase expert trajectories from 100 to 300 (ablation) | CA-ICRL at 80% confidence · Blocked Walker and other tested MuJoCo environments | rewards | Helped |
| Increase expert trajectories from 100 to 300 (ablation) | CA-ICRL at 80% confidence · Blocked Walker | constraint violation rate | No clear effect |
| Increase the desired reward threshold from 1000 to 2000 (ablation) | CA-ICRL at 80% confidence · Blocked Walker | minimum expert trajectories required | Hurt |
| Use CA-ICRL instead of VICRL | ICRL · Biased Pendulum with varied expert trajectory counts | constraint violation rate | Mixed |
| Use CA-ICRL instead of VICRL | ICRL · MuJoCo expected calibration error | ECE | Helped |
| Increase confidence requirement (ablation) | CA-ICRL · MuJoCo environments | constraint violation rate | Helped |
Surprising findings
- Biased Pendulum remains hard for all ICRL methods, with CA-ICRL's violation rate staying high even though it beats baselines.
Limitations
- Assumes the reward function is known and only the constraint is learned.
- Requires online simulator training; offline imitation policy learning is not addressed.
- Assumes infallible experts and cannot guarantee zero ground-truth constraint violations; computational cost grows with number of expert trajectories.
Experimental setup
Datasets: MuJoCo expert trajectories generated with PPO-Lagrange (blocked half-cheetah, ant, biased pendulum, walker, swimmer) HighD dataset
Benchmarks: Blocked Half-Cheetah Blocked Ant Biased Pendulum Blocked Walker Blocked Swimmer HighD distance constraint HighD velocity constraint
Metrics: constraint violation rate cumulative reward expected calibration error p-value (unpaired 2-sided t-test)
Baselines: BC2L GACL ICRL VICRL
Builds on: Inverse Constrained Reinforcement Learning (ICRL) VICRL PPO-Lagrange maximum entropy IRL
Closest papers by meaning
- Robust Inverse Constrained Reinforcement Learning under Model Misspecification (ICML 2024 · similarity 0.92)
- Learning Constraints from Offline Demonstrations via Superior Distribution Correction Estimation (ICML 2024 · similarity 0.89)
- Provably Efficient Exploration in Inverse Constrained Reinforcement Learning (ICML 2025 · similarity 0.89)
- Simplifying Constraint Inference with Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.87)
- Recent Advancements in Inverse Reinforcement Learning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.