Research map / Reinforcement learning / Visual and representation RL
METRA: Scalable Unsupervised RL with Metric-Aware Abstraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes METRA, an unsupervised RL objective that maximizes Wasserstein dependency between states and skills under a temporal-distance metric, scaling skill discovery to pixel-based Quadruped and Humanoid.
Problem
Unsupervised RL struggles to scale: pure exploration cannot cover huge state spaces, and mutual-information skill discovery is metric-agnostic and yields limited coverage, especially in pixel-based high-dimensional environments.
Approach
Cover a compact latent space metrically connected to the state space via temporal distance: maximize a Wasserstein dependency measure with a 1-Lipschitz (temporal-distance) representation phi(s) and linear skills z, with reward (phi(s')-phi(s))^T z optimized by SAC under a dual-constrained Lipschitz constraint.
Key results
- METRA is the only method that discovers diverse locomotion skills in pixel-based Quadruped and Humanoid, per qualitative comparison across 11 baselines.
- METRA achieves the best policy coverage in most domains versus DIAYN, DADS, CIC, and LSD, with 8 seeds and 95% confidence intervals.
- On five goal-reaching downstream tasks, METRA achieves the best performance on all five versus LEXA, DIAYN, and LSD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use METRA instead of prior skill-discovery methods | Unsupervised skill discovery · Five benchmark environments | policy coverage | Helped |
| Use METRA instead of prior skill-discovery methods | Unsupervised skill discovery · Five downstream tasks | downstream task performance | Helped |
| Use METRA instead of prior unsupervised goal-reaching methods | Zero-shot goal-reaching · Five goal-reaching downstream tasks | goal-reaching performance | Helped |
| Use METRA instead of prior methods for pixel-based locomotion skill discovery | Unsupervised skill discovery · Pixel-based Quadruped and Humanoid | locomotion skill diversity | Helped |
| Use METRA instead of pure exploration methods | Unsupervised exploration · Most evaluated environments | total state coverage | Helped |
| Increase latent-space capacity (ablation) | METRA · Ant and HalfCheetah | skill diversity | Helped |
Surprising findings
- Pure exploration methods fail to cover the x-y state space even in state-based Ant and HalfCheetah.
- Mutual-information skill discovery methods, even with exploration bonus (CIC), fail to explore large portions of the state space.
Limitations
- Symmetric Euclidean embedding of asymmetric temporal distance makes abstraction conservative; quasimetric extension left to future work.
- Simplified objective (psi(z)=z) restricts behaviors to linear motion in latent space.
- Relatively small update-to-data ratio and vanilla SAC limit sample efficiency; no evaluation on Atari or non-stationary environments.
Experimental setup
Datasets: MuJoCo HalfCheetah MuJoCo Ant DMC Quadruped (pixel) DMC Humanoid (pixel) pixel-based Kitchen
Benchmarks: Ant HalfCheetah Quadruped Humanoid Kitchen
Metrics: state coverage (x-y bins) policy coverage downstream task return goal-reaching distance task success count
Baselines: DIAYN DADS CIC LSD ICM RND Plan2Explore APT LBS APS LEXA
Builds on: LSD CIC DIAYN DADS VIC
Closest papers by meaning
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.90)
- 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities (NeurIPS 2025 · similarity 0.88)
- PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.87)
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.87)
- XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.