Research map / Reinforcement learning / Visual and representation RL
Robust Policy Learning via Offline Skill Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DuSkill, an offline skill learning framework that uses a hierarchical domain encoder and guided diffusion decoder to disentangle domain-invariant and domain-variant skill embeddings, enabling skill generation beyond the offline datasets for cross-domain few-shot imitation and online RL on multi-stage Meta-World.
Problem
Skill-based RL and few-shot imitation rely on skills learned from offline datasets, which degrade when the downstream task's domain differs from the source domains of the datasets.
Approach
Disentangle skill embeddings into a domain-invariant and a domain-variant part via a hierarchical encoder, decode them with a classifier-free-guided diffusion decoder, and train only the high-level policy on new domains so that generated skills extend beyond the training data.
Key results
- In few-shot imitation, DuSkill achieves 3.99 reward in source speed domain and 3.81 at speed level 3, versus FIST* at 3.28, with ~7.6% average degradation vs ~25.3% for FIST*.
- In online RL on the speed domain, DuSkill reaches 3.32 on target domains vs 0.36 for SPiRL-c, a reported 89.16% gain over SPiRL-c.
- With 1 to 20 few-shot trajectories, DuSkill drops only 4.89% while FIST drops 13.96%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DuSkill instead of the strongest listed baseline | Few-shot imitation · Multi-stage Meta-World Speed | achieved reward: 3.51 ± 0.12 -> 3.83 ± 0.06 (Level 2); 3.28 ± 0.19 -> 3.81 ± 0.06 (Level 3) | Helped |
| Use DuSkill instead of the strongest listed baseline | Few-shot imitation · Multi-stage Meta-World Energy | achieved reward: 3.08 ± 0.19 -> 3.85 ± 0.08 (Source); 2.53 ± 0.25 -> 3.87 ± 0.09 (Level 1); 1.85 ± 0.34 -> 3.71 ± 0.10 (Level 2); 2.02 ± | Helped |
| Use DuSkill instead of the strongest listed baseline | Few-shot imitation · Multi-stage Meta-World Wind | achieved reward: 4.00 ± 0.00 -> 3.98 ± 0.02 (Source); 3.89 ± 0.15 -> 3.78 ± 0.13 (Level 1); 3.24 ± 0.24 -> 3.48 ± 0.21 (Level 2); 3.04 ± | Mixed |
| Use DuSkill instead of SPiRL-c | Online RL adaptation · Speed domain Target | achieved reward: 0.36 ± 0.02 -> 3.32 ± 0.20 | Helped |
| Use DuSkill instead of SPiRL-c | Online RL adaptation · Speed domain Source | achieved reward: 4.00 ± 0.00 -> 4.00 ± 0.00 | No clear effect |
| Add hierarchical embeddings and guided diffusion decoder (ablation) | DU naive-diffusion variant · Multi-stage Meta-World Speed, Energy, and Wind Level 3 | achieved reward: 3.59 ± 0.19 -> 3.81 ± 0.06; 3.16 ± 0.35 -> 3.67 ± 0.14; 3.19 ± 0.17 -> 3.44 ± 0.18 | Helped |
| Add guided diffusion decoder to hierarchical embeddings (ablation) | HDU · Multi-stage Meta-World Speed, Energy, and Wind Level 3 | achieved reward: 3.49 ± 0.68 -> 3.81 ± 0.06; 3.39 ± 0.36 -> 3.67 ± 0.14; 3.08 ± 0.36 -> 3.44 ± 0.18 | Helped |
Surprising findings
- Fine-tuning the entire model (SPiRL-c*) sometimes underperforms fine-tuning only the high-level policy (SPiRL-c), attributed to covariate shift from few samples.
Limitations
- Evaluated only on a single simulated benchmark (multi-stage Meta-World) with rule-based expert data.
- Cross-domain generalization limited to reward and dynamics variations; authors note entirely different tasks, embodiments, or simulators remain future work.
Experimental setup
Datasets: multi-stage Meta-World expert datasets (rule-based heuristic experts)
Benchmarks: Multi-stage Meta-World
Metrics: achieved reward performance degradation across domain levels sample efficiency
Baselines: BC SPiRL SPiRL-c FIST BC+SAC
Builds on: SPiRL FIST DDPM classifier-free guidance soft actor-critic
Closest papers by meaning
- In-Context Policy Adaptation via Cross-Domain Skill Diffusion (AAAI 2025 · similarity 0.90)
- Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.89)
- LLM-based Skill Diffusion for Zero-shot Policy Adaptation (NeurIPS 2024 · similarity 0.89)
- Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment (ICLR 2025 · similarity 0.89)
- Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.