Research map / Label-efficient and robust learning / Self-supervised and contrastive
Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reinterprets METRA as a contrastive mutual-information skill learning method and proposes Contrastive Successor Features (CSF), a simpler MISL algorithm that matches METRA on continuous-control skill discovery benchmarks.
Problem
Whether the strong performance of METRA (Wasserstein-based skill learning) can be explained within the established mutual information skill learning (MISL) framework, and whether MISL can yield simple, competitive unsupervised skill discovery in continuous control.
Approach
Shows METRA's representation objective approximates a contrastive (InfoNCE-like) lower bound on I(S,S';Z) and its actor objective is an MI lower bound minus an anti-exploration term (an information-bottleneck interpretation). Builds CSF: contrastive representation learning with a scaled negative term, intrinsic reward from the positive term only, and successor-feature critics replacing scalar Q-fu
Key results
- CSF matches METRA on state coverage in HalfCheetah and Humanoid exploration.
- METRA and CSF perform best on Humanoid zero-shot goal reaching, with METRA slightly better there; all methods similar on HalfCheetah.
- Replacing METRA's representation loss with the contrastive loss (METRA-C) fully recovers METRA's performance; alternative critic parameterizations (MLP, Gaussian, Laplacian kernels) hurt CSF performance severely.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace METRA's representation objective with a contrastive loss (ablation) | METRA · Ant exploration | state coverage | No clear effect |
| use the information-bottleneck intrinsic reward instead of directly maximizing mutual information (ablation) | CSF · Ant exploration | state coverage | Helped |
| include the anti-exploration term in the intrinsic reward (ablation) | CSF · Ant exploration | state coverage | Hurt |
| use the inner-product transition-difference critic parameterization (ablation) | CSF · Ant exploration | state coverage | Helped |
| use an MLP(s, s')^T z critic parameterization instead of the inner product of transition differences and skills (ablation) | CSF · Ant exploration | state coverage | Hurt |
| use a Gaussian-kernel critic parameterization instead of the inner-product parameterization (ablation) | CSF · Ant exploration | state coverage | Hurt |
| use a Laplacian-kernel critic parameterization instead of the inner-product parameterization (ablation) | CSF · Ant exploration | state coverage | Hurt |
| add a coefficient of 5 to scale the contrastive objective's negative term (ablation) | CSF · skill-learning tasks | performance | Helped |
| use CSF rather than METRA | skill-learning methods · HalfCheetah and Humanoid exploration | state coverage | No clear effect |
| use CSF rather than METRA | skill-learning methods · Humanoid zero-shot goal reaching | staying time fraction | Mixed |
| use CSF rather than METRA | skill-learning methods · HumanoidGoal and HalfCheetahHurdle hierarchical control | discounted return | No clear effect |
Surprising findings
- Learned METRA representations do not satisfy the norm constraint for every transition; the average squared norm of differences converges to about 0.988 with spread, so the constraint holds only in expectation.
- Using the mutual information (including the anti-exploration term) as the reward instead of the information bottleneck causes CSF to fail to discover skills.
Limitations
- Unclear how to scale CSF to more complex environments such as Craftax or VIMA with partial observability, stochasticity, and discrete actions.
- Performance is comparable to METRA rather than clearly superior; results vary by task.
- Pretraining on large real-world datasets (e.g., BridgeData V2, YouCook2) remains open.
Experimental setup
Datasets: Ant HalfCheetah Humanoid Quadruped Kitchen Robobin (LEXA)
Benchmarks: Ant HalfCheetah Humanoid Quadruped Kitchen Robobin HumanoidGoal HalfCheetahHurdle
Metrics: unique (x,y) state coverage staying time fraction for zero-shot goal reaching hierarchical control return
Baselines: METRA CIC DIAYN DADS VISR
Builds on: METRA InfoNCE SAC VISR CIC
Closest papers by meaning
- Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization (NeurIPS 2026 · similarity 0.88)
- Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Skill-aware Mutual Information Optimisation for Zero-shot Generalisation in Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Robust Policy Learning via Offline Skill Diffusion (AAAI 2024 · similarity 0.86)
- Efficient Skill Discovery via Regret-Aware Optimization (ICML 2025 · similarity 0.86)
- Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.