Research map / Reinforcement learning / Imitation learning
GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GO-DICE, an offline imitation learning method that infers option (sub-task) hierarchies from expert and imperfect demonstrations and learns goal-conditioned hierarchical policies via stationary distribution correction estimation, evaluated on Fetch pick-and-place tasks in MuJoCo.
Problem
Offline imitation learning struggles with long-horizon goal-conditioned manipulation tasks and requires retraining when task goals change; existing DICE methods do not model sub-task structure.
Approach
Combines DICE-style stationary distribution matching (DemoDICE-style KL regularization with imperfect demos) with goal-conditioned, option-aware hierarchical policies (high-level option transition policy and low-level action policy), with Viterbi-style iterative option segmentation and optional expert sub-task annotations.
Key results
- On PnP tasks GO-DICE outperforms BC, GoFAR and g-DemoDICE, with the gap widening as object count increases; the paper reports GO-DICE performance degrades on PnP-3.
- GO-DICE-Semi with expert sub-task annotations (E3 labels) achieves higher returns than unsupervised GO-DICE, and with 10 annotated expert demos on PnP-2 matches unsupervised variants using more expert demos.
- Zero-shot transfer: low-level policy trained on PnP-1 gets returns of 2.6±0.49 on PnP-2 and 3.2±1.13 on PnP-3, close to policies trained directly (2.9±0.83, 3.5±0.45).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GO-DICE instead of the offline IL baselines | Offline IL on Fetch Pick-and-Place tasks · PnP-2 and PnP-3 | cumulative reward | Helped |
| Add expert-provided sub-task labels (ablation) | GO-DICE-Semi · Fetch Pick-and-Place tasks | cumulative reward | Helped |
| Increase the number of unsupervised options to K=6 (ablation) | GO-DICE · PnP-2 | cumulative reward | Hurt |
| Increase the number of unsupervised options to K=9 (ablation) | GO-DICE · PnP-3 | cumulative reward | No clear effect |
| Use expert-curated fine-grained labels for primitive and object (E3) (ablation) | GO-DICE-Semi · Fetch Pick-and-Place tasks | cumulative reward | Helped |
| Increase the share of expert demonstrations while keeping the total demonstration count constant (ablation) | GO-DICE-Semi · PnP-2 and PnP-3 | cumulative reward | Helped |
| Use the low-level policy trained on PnP-1 instead of one trained directly on PnP-2 | GO-DICE · PnP-2 | cumulative reward: 2.9\pm 0.83 -> 2.6\pm 0.49 | No clear effect |
| Use the low-level policy trained on PnP-1 instead of one trained directly on PnP-3 | GO-DICE · PnP-3 | cumulative reward: 3.5\pm 0.45 -> 3.2\pm 1.13 | No clear effect |
| Use GO-DICE with K=3 rather than GO-DICE-Semi with K=9 and expert E3 labels (ablation) | GO-DICE on Stack-3 · Stack-3 | cumulative reward | No clear effect |
Surprising findings
- Unsupervised GO-DICE with more options does not monotonically improve; it leaves several options unused (e.g., omits options in PnP-3), suggesting fewer effective options than specified.
- Zero-shot reuse of a low-level policy from the single-object task performs comparably to policies trained on the target multi-object tasks.
Limitations
- Performance gain over baselines shrinks as the number of objects increases; PnP-3 performance diminishes.
- Evaluation limited to robotics manipulation domain.
- Number of options K is unknown and must be tuned as a hyperparameter; more options do not guarantee better performance.
Experimental setup
Datasets: expert demonstrations generated by a hand-crafted/trained expert policy in Fetch Pick-and-Place imperfect demonstrations from noisy expert and random policies
Benchmarks: Fetch Pick-and-Place (PnP-1, PnP-2, PnP-3) Stack-3
Metrics: cumulative reward (number of successful pick/place, averaged fraction of time per option
Baselines: Behavioral cloning GoFAR g-DemoDICE
Builds on: DemoDICE GoFAR options framework Hindsight Experience Replay
Closest papers by meaning
- Zero-Shot Offline Imitation Learning via Optimal Transport (ICML 2025 · similarity 0.90)
- OLLIE: Imitation Learning from Offline Pretraining to Online Finetuning (ICML 2024 · similarity 0.90)
- Learning from Demonstrations via Capability-Aware Goal Sampling (NeurIPS 2025 · similarity 0.90)
- Offline Actor-Critic Reinforcement Learning Scales to Large Models (ICML 2024 · similarity 0.90)
- Imitation Learning from Observation with Automatic Discount Scheduling (ICLR 2024 · similarity 0.89)
- SEABO: A Simple Search-Based Method for Offline Imitation Learning (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.