Research map / Reinforcement learning / Visual and representation RL
Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the successor distance, a temporal distance built from contrastive successor features via a change of variables that provably satisfies the triangle inequality in stochastic MDPs, and learns it via Contrastive Metric Distillation for offline goal-reaching RL.
Problem
Temporal distances used for goal-reaching fail to satisfy the triangle inequality in stochastic settings, which hurts generalization and shortest-path finding.
Approach
Define a successor distance as the log-ratio of discounted state occupancies (g to g versus s to g, minimized over policies), prove it is a quasimetric, and learn it either by a one-step contrastive objective with a quasimetric network or a two-step distillation of contrastive features into an MRN quasimetric.
Key results
- On AntMaze umaze, CMD 2-step reaches 97.0 ± 0.4% success versus 76.8 ± 2.3% for QRL and 79.8 ± 1.6% for CRL (CPC).
- On 2D navigation, the method reaches about 80% of goals while the best prior method reaches around 50%.
- CMD 1-step achieves 68.0 ± 2.1% on large-play and 74.5 ± 2.3% on large-diverse, outperforming all baselines on those tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill contrastive features into a quasimetric architecture | Contrastive RL on synthetic 2D navigation · Synthetic 2D navigation | success rate: approximately 50% -> approximately 80% | Helped |
| Use CMD 1-step instead of QRL | Goal-conditioned policy on AntMaze · AntMaze medium-play | success rate: 76.5 \pm 2.1 -> \mathbf{78.0 \pm 4.0} | Helped |
| Use CMD 1-step instead of QRL | Goal-conditioned policy on AntMaze · AntMaze medium-diverse | success rate: 73.4 \pm 1.9 -> \mathbf{83.0 \pm 3.1} | Helped |
| Use CMD 1-step instead of QRL | Goal-conditioned policy on AntMaze · AntMaze large-play | success rate: 52.9 \pm 2.8 -> \mathbf{68.0 \pm 2.1} | Helped |
| Use CMD 1-step instead of QRL | Goal-conditioned policy on AntMaze · AntMaze large-diverse | success rate: 51.5 \pm 3.8 -> \mathbf{74.5 \pm 2.3} | Helped |
| Use CMD 2-step instead of QRL | Goal-conditioned policy on AntMaze · AntMaze umaze | success rate: 76.8 \pm 2.3 -> \mathbf{97.0 \pm 0.4} | Helped |
| Use CMD 2-step instead of QRL | Goal-conditioned policy on AntMaze · AntMaze umaze-diverse | success rate: 80.1 \pm 1.3 -> \mathbf{90.5 \pm 1.4} | Helped |
| Use CMD 2-step instead of QRL | Goal-conditioned policy on AntMaze · AntMaze large-play | success rate: 52.9 \pm 2.8 -> 59.2 \pm 1.8 | Helped |
| Use CMD 2-step instead of QRL | Goal-conditioned policy on AntMaze · AntMaze large-diverse | success rate: 51.5 \pm 3.8 -> 63.6 \pm 1.9 | Helped |
Surprising findings
- CMD 1-step is best on several AntMaze variants but CMD 2-step is best only on umaze, and it trails QRL and CRL on medium-play and medium-diverse for 2-step.
Limitations
- Theoretical results require discrete state spaces, while experiments use continuous settings.
- Distance may be infinite in non-ergodic settings.
- AntMaze comparisons restricted to small network sizes.
Experimental setup
Datasets: AntMaze suite synthetic 2D navigation trajectories
Benchmarks: AntMaze umaze AntMaze medium AntMaze large synthetic 2D navigation
Metrics: success rate standard error
Baselines: DDPG Quasimetric RL Contrastive RL (CPC) Contrastive RL (NCE) GCBC IQL
Builds on: Contrastive RL Quasimetric RL Metric Residual Network infoNCE Contrastive learning of successor features
Closest papers by meaning
- Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations (NeurIPS 2025 · similarity 0.89)
- Temporal Difference Flows (ICML 2025 · similarity 0.86)
- Contrastive Representations for Temporal Reasoning (NeurIPS 2025 · similarity 0.86)
- ContraDiff: Planning Towards High Return States via Contrastive Learning (ICLR 2025 · similarity 0.86)
- Hitting Time Isomorphism for Multi-Stage Planning with Foundation Policies (NeurIPS 2026 · similarity 0.86)
- Learning Successor Features the Simple Way (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.