Research map / Reinforcement learning / Imitation learning
Causal Imitation for Markov Decision Processes: a Partial Identification Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that imitation under unobserved confounding in MDPs is generally impossible, and proposes two partial-identification GAIL variants (CAIL-R, CAIL-T) for settings where only the transition or only the reward is non-identifiable.
Problem
Imitation learning from expert demonstrations confounded by unobserved variables in MDPs, where the transition distribution or reward function is not identifiable from observational data, so the imitator may not match expert performance.
Approach
Uses partial identification (bounds on transition and reward from confounded data) and folds the worst-case bounds into GAIL's occupancy-matching objective: CAIL-R reweights the discriminator signal by the propensity score P(x|s) when rewards are confounded; CAIL-T generates imitator rollouts from a worst-case transition model found by a linear program when transitions are confounded.
Key results
- In 1000 random confounded MDPs, all imitators' performance gaps were negative (worse than expert), supporting the impossibility theorem.
- In the driving MDP with confounded reward, CAIL consistently outperforms BC and GAIL in average return over 100 trajectories.
- In the MIMIC-III-based medical treatment setting, the proposed framework performs best among strategies while BC and IRL fail to match expert performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CAIL-R's propensity-weighted discriminator signal instead of standard GAIL | imitation learning with confounded reward and identifiable transition (MDPobs-Y) · Driving | average return | Helped |
| use CAIL-T's worst-case occupancy measure for policy learning instead of standard GAIL | imitation learning with confounded transition and identifiable reward (MDPobs-S) · MIMIC-III | expected return | Helped |
| use CAIL rather than BC or GAIL | imitation learning when reward and transition are both confounded · MDPobs random instances | performance gap | Helped |
| learn an imitation policy from demonstrations when reward and transition are both confounded | imitation learning · MDPobs random instances | performance gap | Hurt |
Surprising findings
- Expert performance is generally not imitable when both transition and reward are confounded; no policy is guaranteed to match the expert across all compatible MDPs.
- In the random-MDP experiment, CAIL still underperforms the expert (gap of -1.9019) though it beats BC and GAIL.
Limitations
- Impossibility result implies no guarantee of expert performance when both transition and reward are confounded.
- Relies on parametric reward family and bounded observational constraints; the worst-case transition requires iterative linear programs.
- Experiments are small-scale simulations, and source code release was only promised for camera-ready.
Experimental setup
Datasets: MIMIC-III synthetic MDPs with binary variables simulated driving environment
Benchmarks: MIMIC-III medical treatment simulation simulated driving MDP 1000 random discrete MDPs
Metrics: expected return performance gap vs expert
Baselines: BC GAIL
Builds on: GAIL Bellman equation partial identification bounds (Manski/Balke-Pearl style natural bounds)
Closest papers by meaning
- Causal Imitation Learning under Expert-Observable and Expert-Unobservable Confounding (ICLR 2026 · similarity 0.92)
- Causal Imitation Learning With Unobserved Confounders (NeurIPS 2020 · similarity 0.92)
- Cross-domain Imitation from Observations (ICML 2021 · similarity 0.92)
- Toward the Fundamental Limits of Imitation Learning (NeurIPS 2020 · similarity 0.91)
- Imitation Learning from Observation with Automatic Discount Scheduling (ICLR 2024 · similarity 0.90)
- Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.