Research map / Reinforcement learning / Visual and representation RL
Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GO-Skill, which learns a discrete library of goal-oriented skills via vector quantization from task-mixed offline data, then trains a Prompt-DT high-level policy to select skills, evaluated on MetaWorld MT30/MT50.
Problem
Offline multi-task RL struggles to share knowledge across tasks when trained on task-mixed datasets with mixed-quality (random to expert) trajectories.
Approach
Extract reusable skills as H-step sub-trajectories represented by goal embeddings (state differences rather than actions or rewards), quantize them into a discrete codebook, enhance rare skills with resampling, and learn a hierarchical skill-selecting Transformer policy with focal loss.
Key results
- On MT50 Near-Optimal, the full GO-Skill reaches 3159.1 average return (±71.0) over 5 seeds, and 2123.8 on Sub-Optimal.
- Replacing the goal-based skill encoder with an action-based one drops MT50 Near-Optimal return from 3159.1 to 2797.5.
- Removing focal loss drops Near-Optimal return to 3048.4 and removing resampling drops it to 2962.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add reached-goal history to the skill decoder (ablation) | GO-Skill · MetaWorld MT50 Near-Optimal and Sub-Optimal | average episode return: $2952.1_{\pm 92.3}$ -> $3159.1_{\pm 71.0}$; $2041.5_{\pm 83.5}$ -> $2123.8_{\pm 78.1}$ | Helped |
| Use a discrete VQ skill space instead of a continuous skill space (ablation) | GO-Skill · MetaWorld MT50 Near-Optimal and Sub-Optimal | average episode return: $2905.0_{\pm 92.7}$ -> $3159.1_{\pm 71.0}$; $1999.0_{\pm 82.4}$ -> $2123.8_{\pm 78.1}$ | Helped |
| Use goal-based rather than action-subsequence-based skill encoding (ablation) | GO-Skill · MetaWorld MT50 Near-Optimal and Sub-Optimal | average episode return: $2797.5_{\pm 141.7}$ -> $3159.1_{\pm 71.0}$; $1947.9_{\pm 126.2}$ -> $2123.8_{\pm 78.1}$ | Helped |
| Add focal loss to skill-based policy training (ablation) | GO-Skill · MetaWorld MT50 Near-Optimal and Sub-Optimal | average episode return: $3048.4_{\pm 90.9}$ -> $3159.1_{\pm 71.0}$; $2092.7_{\pm 71.0}$ -> $2123.8_{\pm 78.1}$ | Helped |
| Add uniform skill-class resampling during skill enhancement (ablation) | GO-Skill · MetaWorld MT50 Near-Optimal and Sub-Optimal | average episode return: $2962.3_{\pm 89.4}$ -> $3159.1_{\pm 71.0}$; $2012.9_{\pm 109.2}$ -> $2123.8_{\pm 78.1}$ | Helped |
Surprising findings
- Continuous skill embeddings performed worse than discrete skill indices (VQ ablation: 2905.0 vs 3159.1 Near-Optimal), which the authors attribute to discrete skills being easier to learn to use.
Limitations
- Performance depends on a predefined skill horizon H and codebook size M.
- Goal representation is currently a state difference, with more robust goal-oriented representations left for future work.
- Dynamic skill set extension and adaptive skill horizon are not addressed.
Experimental setup
Datasets: MetaWorld SAC-Replay near-optimal dataset MetaWorld SAC-Replay sub-optimal dataset
Benchmarks: MetaWorld MT30 MetaWorld MT50 MetaWorld ML45 fine-tuning
Metrics: average episode return
Baselines: MTDT MH-MTDT PromptDT MH-PromptDT MTDIFF-P HarmoDT
Builds on: Prompt-DT Decision Transformer VQ-VAE focal loss
Closest papers by meaning
- Robust Policy Learning via Offline Skill Diffusion (AAAI 2024 · similarity 0.89)
- Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperation (ICLR 2025 · similarity 0.89)
- GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation (AAAI 2024 · similarity 0.88)
- SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution (CVPR 2024 · similarity 0.88)
- Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment (ICLR 2025 · similarity 0.88)
- STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.