Research map / Reinforcement learning / Visual and representation RL
Decoupling Meta-Reinforcement Learning with Gaussian Task Contexts and Skills
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DCMRL, an offline meta-RL framework that contrastively regularizes Gaussian task contexts and clusters task-context and skill Gaussians with GQ-VAE codebooks to improve generalization to unseen tasks in maze navigation and kitchen manipulation.
Problem
Offline meta-RL agents learn prior experience (task contexts and skills) that generalizes poorly to unseen target tasks because exploration and learning of continuous latent spaces are coupled, and task/skill structure is not modeled.
Approach
Model task contexts and skills as Gaussian distributions, apply a triplet contrastive loss to task contexts, and use a Gaussian quantization VAE (GQ-VAE) with learnable codebooks to cluster each latent space, decoupling exploration of the continuous space from learning of discrete cluster centers.
Key results
- DCMRL solves nearly 90% of unseen maze target tasks within a few episodes and nearly 100% eventually, outperforming SiMPL, SPiRL, MTRL, and PEARL baselines.
- In kitchen manipulation DCMRL completes over three of four subtasks, versus baselines that reach fewer subtasks.
- Ablations show removing contrastive learning or GQ-VAE lowers sample efficiency and performance; DCMRL with VQ-VAE fails to adapt in maze navigation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DCMRL instead of prior meta-RL methods | Unseen target-task adaptation · Maze navigation | adaptation performance and sample efficiency | Helped |
| Use DCMRL instead of prior meta-RL methods | Unseen target-task adaptation · Kitchen manipulation | adaptation performance | Helped |
| Remove contrastive learning (ablation) | DCMRL · Kitchen manipulation | sample efficiency and performance | Hurt |
| Remove GQ-VAE (ablation) | DCMRL · Kitchen manipulation | sample efficiency and performance | Hurt |
| Replace GQ-VAE with VQ-VAE (ablation) | DCMRL · Maze navigation | adaptation performance | Hurt |
| Replace GQ-VAE with VQ-VAE (ablation) | DCMRL · Kitchen manipulation | adaptation performance | Hurt |
| Vary the number of meta-training tasks from 40 to 10 or 20 (ablation) | DCMRL · Maze navigation | adaptation performance | No clear effect |
| Increase the task-context code count to 32 or 64 while keeping 16 skill codes (ablation) | DCMRL · Kitchen manipulation | sample efficiency and performance | Hurt |
| Increase the skill code count to 32 or 64 while keeping 16 task-context codes (ablation) | DCMRL · Kitchen manipulation | sample efficiency and performance | Hurt |
| Increase both task-context and skill code counts to 64 or 128 (ablation) | DCMRL · Kitchen manipulation | sample efficiency and performance | No clear effect |
Surprising findings
- DCMRL trained on a misaligned meta-training task distribution initially underperforms but eventually reaches similarly superior performance.
Limitations
- Performance sensitive to number of codes in GQ-VAE
- Evaluated only on two simulated environments (maze navigation, kitchen manipulation)
Experimental setup
Datasets: maze navigation offline dataset (planner-generated trajectories) Franka kitchen teleoperated dataset (600 sequences)
Benchmarks: maze navigation Franka kitchen manipulation
Metrics: task success / subtasks completed sample efficiency (episodes to adapt)
Baselines: SiMPL SAC SPiRL PEARL PEARL-ft MTRL VQ-VAE variant of DCMRL
Builds on: SPiRL PEARL SiMPL VQ-VAE
Closest papers by meaning
- MetaCARD: Meta-Reinforcement Learning with Task Uncertainty Feedback via Decoupled Context-Aware Reward and Dynamics Components (AAAI 2024 · similarity 0.92)
- Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learning (ICLR 2025 · similarity 0.91)
- Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Meta-Reinforcement Learning via Exploratory Task Clustering (AAAI 2024 · similarity 0.90)
- Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations (AAAI 2024 · similarity 0.90)
- Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.