Research map / Reinforcement learning / Imitation learning
Learning Constraints from Offline Demonstrations via Superior Distribution Correction Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ICSDICE, an offline inverse constrained RL method that infers sparse hard constraints from expert and suboptimal offline demonstrations via a DICE-style dual optimization over superior distributions.
Problem
Inverse constrained RL normally needs online environment interaction, which is dangerous or expensive in many real applications; the paper addresses recovering safety constraints purely from a fixed offline dataset.
Approach
Alternates a forward offline DICE dual problem (chi-square regularized, with discounted visitation distribution and a clipped correction term) that finds superior distributions with rewards above the expert, and a backward step that assigns high cost to the support of those distributions while keeping expert-visited pairs at zero cost, encouraging sparse constraints.
Key results
- On MuJoCo with 100% expert data, ICSDICE keeps cost below 5% violation on all five tasks (e.g., Ant(Blocked) cost 0.01 with reward 3,073 vs. OptiDICE-c 3,070).
- ICSDICE achieves AUC 0.94 on Ant(LS) and 0.91 on Ant(Blocked) in cost identification, far above OptiDICE-c (0.60 and 0.54).
- On HighD speed-constraint task, ICSDICE gets return 10.7 and cost 6.5% vs. OptiDICE-c return -1.6 and cost 12%; image-based car distance AUC 0.75 vs 0.69.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ICSDICE instead of OptiDICE-c | ICSDICE · HighD speed constraint | AUC: 0.76 ± 0.02 -> 0.81 ± 0.01 | Helped |
| use ICSDICE instead of OptiDICE-c | ICSDICE · HighD image distance constraint | AUC: 0.69 -> 0.75 | Helped |
| replace discounted visitation distribution with empirical distribution (ablation) | ICSDICE · Ant(LS) | AUC: 0.94±0.01 -> 0.98±0.01 | Helped |
| replace discounted visitation distribution with empirical distribution (ablation) | ICSDICE · Ant(Blocked) | AUC: 0.91±0.01 -> 0.88±0.01 | Hurt |
| replace discounted visitation distribution with empirical distribution (ablation) | ICSDICE · Hopper(Blocked) | AUC: 0.63±0.04 -> 0.63±0.03 | No clear effect |
| replace discounted visitation distribution with empirical distribution (ablation) | ICSDICE · HalfCheetah(Obstacle) | AUC: 0.98±0.02 -> 0.77±0.05 | Hurt |
| replace discounted visitation distribution with empirical distribution (ablation) | ICSDICE · Walker(LS) | AUC: 0.93±0.02 -> 0.86±0.00 | Hurt |
| replace discounted visitation distribution with empirical distribution (ablation) | ICSDICE · MuJoCo tasks | reward | Mixed |
| remove the constraint learning step (ablation) | ICSDICE · MuJoCo tasks | cost | Hurt |
Surprising findings
- OptiDICE-Constraint performs comparably to ICSDICE on policy recovery, but its learned constraints have much lower cost identification accuracy due to biased distribution estimation.
- Removing the constraint learning step leaves policy reward high but violation rate much higher (e.g., Ant(Blocked) cost 0.21 vs 0.01).
Limitations
- Does not account for uncertainty in the offline dataset; unvisited states are not handled explicitly.
- No theoretical analysis of convergence rate or sample efficiency is provided.
Experimental setup
Datasets: MuJoCo expert demonstrations (Ant, HalfCheetah, Hopper, Walker constrained variants) HighD commonroad-RL 7x7 grid-world offline dataset
Benchmarks: MuJoCo constrained tasks (Ant(Blocked), Ant(LS), HalfCheetah(Obstacle), Hopper(Blocked), Walker(LS)) HighD speed constraint commonroad-RL car distance constraint 7x7 grid-world constraints
Metrics: cumulative reward cost / constraint violation rate AUC of ROC for learned cost feasible cumulative reward
Baselines: LS-IQ SMODICE Behavior Cloning OptiDICE-Constraint SMODICE-Constraint Offline IL Offline IL + rewards
Builds on: OptiDICE Inverse Constrained Reinforcement Learning (Malik et al., 2021) SMODICE Inverse Q-learning DICE
Closest papers by meaning
- Robust Inverse Constrained Reinforcement Learning under Model Misspecification (ICML 2024 · similarity 0.92)
- Confidence Aware Inverse Constrained Reinforcement Learning (ICML 2024 · similarity 0.89)
- Provably Efficient Exploration in Inverse Constrained Reinforcement Learning (ICML 2025 · similarity 0.89)
- Simplifying Constraint Inference with Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Recent Advancements in Inverse Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.