Research map / Reinforcement learning / Visual and representation RL
Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Temporal Metric Distillation (TMD), an offline GCRL method that combines contrastive Monte Carlo temporal distance learning with quasimetric (MRN) architecture and two transition-level invariances to recover optimal goal-reaching distances in stochastic settings.
Problem
Offline goal-conditioned RL needs temporal distance representations that both stitch shorter paths than demonstrated and capture long-horizon distances with low variance, which existing contrastive, quasimetric, and TD methods do not jointly achieve, especially under stochastic dynamics and suboptim
Approach
Initialize a quasimetric distance with a backward NCE contrastive (Monte Carlo) objective, then enforce action invariance and a SARSA-style temporal Bellman invariance (with a Bregman divergence D_T) on top of a triangle-inequality MRN architecture, which converges to the optimal successor distance.
Key results
- TMD achieves 29.3% on pointmaze_teleport_stitch versus CRL and QRL, outperforming both by over 3x, and 68.5% on humanoidmaze_medium_stitch versus 36.2% for CRL.
- TMD obtains 49.6% on antmaze_teleport_explore versus 19.5% for CRL and 2.3% for QRL.
- Ablations on pointmaze_teleport_stitch show removing the contrastive loss drops success to 9.8% and removing the T loss drops it to 18.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove T-invariance loss (ablation) | TMD · pointmaze_teleport_stitch | success rate (%): 29.3(±2.2) -> 18.5(±2.1) | Hurt |
| remove action-invariance loss (ablation) | TMD · pointmaze_teleport_stitch | success rate (%): 29.3(±2.2) -> 13.3(±2.9) | Hurt |
| remove contrastive loss (ablation) | TMD · pointmaze_teleport_stitch | success rate (%): 29.3(±2.2) -> 9.8(±1.7) | Hurt |
| do not stop gradients in LT (ablation) | TMD · pointmaze_teleport_stitch | success rate (%): 29.3(±2.2) -> 18.7(±1.8) | Hurt |
| replace DT divergence with Dℓ2 (ablation) | TMD · antmaze_teleport_stitch | success rate (%): 29.3(±2.2) -> 16.1(±1.9) | Hurt |
| replace DT divergence with DBCE (ablation) | TMD · antmaze_teleport_stitch | success rate (%): 29.3(±2.2) -> 15.1(±1.9) | Hurt |
| use TMD instead of the strongest listed baseline | offline GCRL · humanoidmaze_large_stitch | success rate (%): 9.3(±0.7) -> 23.0(±1.5) | Helped |
| use TMD instead of the strongest listed baseline | offline GCRL · antmaze_large_stitch | success rate (%): 29.0(±2.3) -> 37.3(±2.7) | Helped |
| use TMD instead of the strongest listed baseline | offline GCRL · antmaze_teleport_explore | success rate (%): 32.0(±0.6) -> 49.6(±1.5) | Helped |
| use TMD instead of the strongest listed baseline | offline GCRL · pointmaze_teleport_stitch | success rate (%): 44.4(±0.7) -> 29.3(±2.2) | Hurt |
| use TMD instead of the strongest listed baseline | offline GCRL · scene_noisy | success rate (%): 26.4(±1.7) -> 19.6(±1.7) | Hurt |
| use TMD instead of the strongest listed baseline | offline GCRL · visual_antmaze_giant_navigate | success rate (%): 47.2(±0.9) -> 40.1(±2.6) | Hurt |
Surprising findings
- Stopping gradients through the temporal invariance loss improves performance (29.3% vs 18.7% without stopgrad).
- Exponentiated Bregman divergence D_T for the temporal backup outperforms squared loss and BCE (29.3% vs 16.1% and 15.1% on antmaze-teleport-stitch).
Limitations
- The invariance weight zeta must be set heuristically per environment and is kept constant in practice rather than annealed.
- Policy extraction still relies on behavioral cloning regularization with per-environment alpha tuning.
- Only the MRN quasimetric architecture was tested; more expressive alternatives like IQE were not explored.
Experimental setup
Datasets: OGBench humanoidmaze OGBench pointmaze OGBench antmaze OGBench scene OGBench visual_antmaze OGBench visual_cube
Benchmarks: OGBench humanoidmaze OGBench pointmaze OGBench antmaze OGBench scene OGBench visual_antmaze OGBench visual_cube
Metrics: success rate (%) standard error across six seeds
Baselines: GCBC GCIQL GCIVL CRL QRL CMD
Builds on: Contrastive Reinforcement Learning Quasimetric Reinforcement Learning Contrastive Metric Distillation Metric Residual Network behavior-constrained deep deterministic policy gradient
Closest papers by meaning
- Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making (ICML 2024 · similarity 0.89)
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.89)
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.89)
- Conservative Offline Goal-Conditioned Implicit V-Learning (ICML 2025 · similarity 0.88)
- Score Models for Offline Goal-Conditioned Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Representation-Based Robustness in Goal-Conditioned Reinforcement Learning (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.