Research map / Reinforcement learning / Visual and representation RL
Skill-aware Mutual Information Optimisation for Zero-shot Generalisation in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Skill-aware Mutual Information (SaMI) and a K-sample estimator SaNCE to train Meta-RL context encoders that distinguish skills, improving zero-shot generalisation on modified MuJoCo and Panda-gym with smaller negative sample spaces.
Problem
Meta-RL context encoders trained with contrastive InfoNCE only separate tasks, not skills, and InfoNCE's K-sample MI estimate is bottlenecked by log K, requiring large sample sizes that are impractical in zero-shot RL generalisation.
Approach
Define SaMI as an interaction-information objective over context embedding, skill and trajectory, and optimise it with SaNCE, a K-sample estimator whose negatives are drawn from the current task across skills (rather than all tasks), reducing the required sample space while remaining a tighter lower bound than InfoNCE.
Key results
- On Panda-gym, SaCCM reaches 0.57±0.26 success on moderate test tasks versus 0.49±0.23 for CCM, with SaTESAC at 0.37±0.34 on extreme tasks versus 0.22±0.21 for TESAC.
- On modified MuJoCo, SaCCM achieves 5215±377 return on Ant extreme test versus 4700±696 for CCM, and SaTESAC reaches 6473 on SlimHumanoid extreme (SaCCM 6473±2001) versus 2887 for CCM.
- SaNCE retains performance with small buffers and contrastive batch sizes (e.g., batch size 12 on MuJoCo), suggesting reduced sensitivity to the log-K curse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SaMI/SaNCE objective to TESAC (SaTESAC) | TESAC · Panda-gym | success rate (test extreme): 0.22 -> 0.37 | Helped |
| Add SaMI/SaNCE objective to TESAC (SaTESAC) | TESAC · Panda-gym | success rate (test moderate): 0.31 -> 0.56 | Helped |
| Add SaNCE to CCM (SaCCM) | CCM · Panda-gym | success rate (test moderate): 0.49 -> 0.57 | Helped |
| Add SaNCE to CCM (SaCCM) | CCM · Panda-gym | success rate (test extreme): 0.29 -> 0.36 | Helped |
| Add SaNCE to CCM (SaCCM) | CCM · MuJoCo Crippled Walker | average return (test extreme): 3098 -> 4824 | Helped |
| Add SaNCE to TESAC (SaTESAC) | TESAC · MuJoCo Crippled Ant | average return (test extreme): 1048 -> 2131 | Helped |
| Add SaNCE to CCM (SaCCM) | CCM · MuJoCo Half-cheetah | average return (test extreme): 1325 -> 1926 | Helped |
| Add SaNCE to TESAC (SaTESAC) | TESAC · MuJoCo Hopper-like tasks | average return | No clear effect |
| Add SaNCE to CCM and TESAC | Panda-gym and MuJoCo · Panda-gym training | success rate: 0.50 -> 0.92 (TESAC); 0.80 -> 0.93 (CCM) | Helped |
| Reduce negative sample space via SaNCE (current-task sampling) vs InfoNCE across all tasks (ablation) | SaNCE vs InfoNCE · Sample-limited analysis (buffer size and contrastive batch size, SlimHumanoid) | average return | No clear effect |
Surprising findings
- SaTESAC and SaCCM do not outperform baselines in Ant, Half-cheetah and Hopper, where only a single skill is needed, showing gains concentrate in multi-skill environments.
Limitations
- Positive and negative skill labels are defined by return ranking, which may be noisy early in training when all returns are low.
- Gains are not uniform; several single-skill environments show no improvement over baselines.
- Evaluated only on simulated benchmarks with independent environmental features; correlated features are left to future work.
Experimental setup
Datasets: Panda-gym (modified) MuJoCo (modified, DOMINO/CaDM extension)
Benchmarks: modified Panda-gym modified MuJoCo
Metrics: success rate average return
Baselines: PEARL TESAC CCM DOMINO CaDM
Builds on: InfoNCE CCM TESAC PEARL SAC
Closest papers by meaning
- AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers (NeurIPS 2024 · similarity 0.90)
- Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity (NeurIPS 2024 · similarity 0.89)
- MetaCARD: Meta-Reinforcement Learning with Task Uncertainty Feedback via Decoupled Context-Aware Reward and Dynamics Components (AAAI 2024 · similarity 0.89)
- Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations (AAAI 2024 · similarity 0.89)
- Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Decoupling Meta-Reinforcement Learning with Gaussian Task Contexts and Skills (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.