Research map / Reinforcement learning / Imitation learning
Mitigating Covariate Shift in Behavioral Cloning via Robust Stationary Distribution Correction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DrilDICE, a distributionally robust behavioral cloning method that constrains the adversarial state distribution to Bellman-flow-feasible stationary distributions via DICE, with a soft-TV f-divergence, to mitigate covariate shift from arbitrarily collected offline expert demonstrations.
Problem
Offline imitation learning where expert demonstration states are drawn from an arbitrary distribution rather than the expert's stationary distribution, causing covariate shift that degrades behavioral cloning and distribution-matching IL methods.
Approach
Replace the unconstrained worst-case BC objective with a minimax over stationary distributions (Bellman flow constraint) regularized by f-divergence to the dataset distribution, solved via DICE-style closed-form weights; a soft-TV f-divergence with a tanh-inverse derivative makes the TV-like weighting tractable.
Key results
- On Four Rooms (p=0.4 manipulation), DrilDICE achieved the highest normalized score in all 8 scenarios, e.g., Room 3 worst-25% of 80.16 vs 55.28 for BC.
- On D4RL rebalanced-by-state settings, DrilDICE outperformed 14 of 15 settings, e.g., walker2d p(D1)=0.1 score 51.6 vs 18.9 for BC.
- In the time-dependent scenario, DrilDICE achieved the highest mean in 11 of 12 settings, e.g., walker2d (1,1) 81.2 vs 29.0 for BC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Bellman-flow-constrained robust weighting rather than the strongest listed baseline | DrilDICE vs. OptiDICE-BC · Four Rooms, marginal room and action manipulations | Normalized score: 94.30 ± 0.41 -> 95.04 ± 0.48; 94.06 ± 0.65 -> 94.44 ± 0.62; 94.20 ± 0.98 -> 95.04 ± 0.86; 94.26 ± 0.54 -> 94.92 ± 0.37; | Helped |
| use Bellman-flow-constrained robust weighting rather than DR-BC | DrilDICE · hopper, walker2d, and halfcheetah rebalanced by state, p(D1)=0.5 | Normalized score: 36.7 ± 3.6 -> 67.1 ± 8.2; 45.1 ± 10.0 -> 73.7 ± 5.4; 26.2 ± 4.9 -> 55.0 ± 3.3 | Helped |
| use robust weighting instead of BC | DrilDICE · hopper, walker2d, and halfcheetah with timestep dependency (1, 1) | Normalized score: 28.9 ± 3.8 -> 45.4 ± 5.1; 29.0 ± 5.3 -> 81.2 ± 5.4; 33.7 ± 3.0 -> 44.2 ± 5.6 | Helped |
| replace soft-TV distance with soft-χ2 divergence (ablation) | DrilDICE · hopper, rebalanced by state, p(D1)=0.5 | Normalized score: 67.1 ± 8.2 -> 60.9 ± 7.6 | Hurt |
| replace soft-TV distance with soft-χ2 divergence (ablation) | DrilDICE · walker2d, timestep dependency (1, 1) | Normalized score: 81.2 ± 5.4 -> 51.9 ± 5.3 | Hurt |
Surprising findings
- Soft-TV f-divergence substantially outperforms soft-chi^2 for DrilDICE on D4RL covariate-shift scenarios (e.g., walker2d p(D1)=0.5: 73.7 vs 30.5).
- Standard BC-based and distribution-matching baselines (DemoDICE, AW-BC, OptiDICE-BC) often fail to outperform plain BC under these shifts.
Limitations
- Does not consider transition shift or noisy expert demonstrations.
- Theoretical guarantees rely on realizability (expert stationary distribution contained in the uncertainty set); no formal theorems for the proposed objective.
- Evaluated only in a toy gridworld and three D4RL locomotion tasks; no real-world application.
Experimental setup
Datasets: D4RL hopper-expert-v2 D4RL walker2d-expert-v2 D4RL halfcheetah-expert-v2 D4RL medium-v2 (supplementary) Four Rooms gridworld expert dataset
Benchmarks: Four Rooms covariate shift scenarios D4RL hopper D4RL walker2d D4RL halfcheetah
Metrics: normalized score worst-25% normalized score target 0-1 loss target MSE
Baselines: BC DemoDICE AW-BC DR-BC OptiDICE-BC
Builds on: DemoDICE OptiDICE DR-BC DICE
Closest papers by meaning
- Mitigating Covariate Shift in Imitation Learning via Offline Data With Partial Coverage (NeurIPS 2021 · similarity 0.95)
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.91)
- How to Leverage Diverse Demonstrations in Offline Imitation Learning (ICML 2024 · similarity 0.90)
- Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift (NeurIPS 2026 · similarity 0.89)
- Imitation from Observations with Trajectory-Level Generative Embeddings (NeurIPS 2026 · similarity 0.88)
- Imitation Learning for Multi-turn LM Agents via On-policy Expert Corrections (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.