Research map / Reinforcement learning / Visual and representation RL
Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Di-SkilL, an RL method that trains a Mixture of Experts of contextual motion primitives with energy-based per-expert context distributions to learn diverse skills via automatic curriculum learning.
Problem
Learning diverse skills in reinforcement learning is hard because standard Gaussian policies capture only a single behavior mode, and automatic curriculum learning over unknown, bounded context distributions with hard discontinuities is difficult.
Approach
Each MoE expert is a contextual motion primitive policy, and each expert has an energy-based per-expert context distribution trained with the policy gradient objective (via PPO) under a KL-regularized maximum-entropy objective, so experts choose their own curricula within the valid context region without environment knowledge or punishment terms.
Key results
- Di-SkilL performs on par with or better than BBRL, LinDi-SkilL and PPO across the evaluated robot tasks, outperforming BBRL on Robot Mini Golf and Box Pushing with Obstacle.
- Ablation on Table Tennis shows disabling automatic curriculum learning (Di-SkilLV2) leads to much smaller final success rate than Di-SkilL.
- Linear-expert variant LinDi-SkilL performs worse on non-Markovian tasks TT-H and MG, indicating non-linear experts are beneficial.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use automatic curriculum learning by sampling preferred context regions for each expert (ablation) | Di-SkilL · Table Tennis | success rate | Helped |
| Disable automatic curriculum learning, using 50 context-parameter samples per expert (ablation) | Di-SkilL · Table Tennis | success rate | Hurt |
| Disable automatic curriculum learning, using 260 context-parameter samples per expert (ablation) | Di-SkilL · Table Tennis | success rate | Hurt |
| Replace nonlinear experts with linear experts (ablation) | Di-SkilL · Table Tennis Hard and Robot Mini Golf | performance | Hurt |
| Replace nonlinear experts with linear experts (ablation) | Di-SkilL · Box Pushing with Obstacle | success rate | Helped |
| Replace nonlinear experts with linear experts (ablation) | Di-SkilL · Hopper Jump | performance | Hurt |
| Use Di-SkilL's automatic curriculum learning and nonlinear experts instead of SVSL's linear experts and Gaussian context distributions | SVSL · Table Tennis | success rate | Helped |
| Use a nonlinear mixture-of-experts policy instead of a single-mode policy | BBRL · Box Pushing with Obstacle | success rate | Helped |
| Use a nonlinear mixture-of-experts policy instead of a single-mode policy | BBRL · 5-Link Reacher | return | Helped |
| Use motion-primitive-based time-correlated exploration instead of PPO exploration | PPO · Box Pushing with Obstacle | success rate | Helped |
| Use a nonlinear policy instead of linear experts (ablation) | LinDi-SkilL · 5-Link Reacher | return | Helped |
| Use a nonlinear policy instead of linear experts (ablation) | LinDi-SkilL · Table Tennis Hard and Robot Mini Golf | performance | Helped |
Surprising findings
- PPO suffers under time-sparse rewards, performing poorly relative to CEPS methods on box pushing and 5-link reacher.
Limitations
- Cannot replan, so small collisions with objects cause task failure
- Sample complexity is high since the method is on-policy
Experimental setup
Benchmarks: Table Tennis Table Tennis Hard 5-Link Reacher Hopper Jump Box Pushing with Obstacle Robot Mini Golf
Metrics: success rate return interquantile mean
Baselines: BBRL SVSL LinDi-SkilL PPO
Builds on: BBRL SVSL ProDMPs PPO
Closest papers by meaning
- Exploration by Learning Diverse Skills through Successor State Representations (NeurIPS 2024 · similarity 0.87)
- Skill Disentanglement in Reproducing Kernel Hilbert Space (AAAI 2025 · similarity 0.87)
- Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.87)
- Robust Policy Learning via Offline Skill Diffusion (AAAI 2024 · similarity 0.87)
- Diffusion-based Curriculum Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Skill-aware Mutual Information Optimisation for Zero-shot Generalisation in Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.