Research map / Reinforcement learning / Visual and representation RL
Episodic Novelty Through Temporal Distance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ETD, an episodic intrinsic reward for sparse-reward CMDPs that uses a learned temporal distance (successor distance quasimetric via contrastive learning) to measure novelty of states relative to episodic memory.
Problem
Exploration in sparse-reward contextual MDPs, where episodes differ, so episodic bonuses based on counts fail in large/noisy state spaces and similarity measures (Euclidean on inverse-dynamics embeddings, reachability likelihoods) poorly capture state novelty.
Approach
Use a learned temporal distance (successor distance, an asymmetric quasimetric) trained with symmetric InfoNCE and an MRN-plus-potential energy parameterization as the similarity metric; the episodic bonus is the minimum temporal distance from the current state to all earlier states in the episode.
Key results
- On ObstructedMaze-Full, ETD reaches near-optimal performance within 20M steps, about double the sample efficiency of NovelD, the strongest baseline.
- With Gaussian noise (variance 0.1) added to MiniGrid DoorKey-16x16 states, NovelD fails completely while ETD maintains strong performance.
- ETD consistently outperforms or matches baselines (PPO, DEIR, NovelD) on MiniWorld Maze and Crafter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ETD temporal-distance episodic bonuses instead of NovelD | PPO · MiniGrid ObstructedMaze-Full | sample efficiency | Helped |
| Use ETD temporal-distance episodic bonuses instead of other exploration methods | PPO · MiniGrid environments | exploration learning efficiency | Helped |
| Use ETD instead of NovelD, E3B, or DEIR | PPO with noisy MiniGrid observations · Noisy MiniGrid | exploration performance | Helped |
| Use ETD instead of PPO without intrinsic rewards | PPO · DMC sparse-reward tasks | performance | Helped |
| Add intrinsic motivation | PPO · HalfCheetahVelSparse | performance | No clear effect |
| Use ETD instead of Euclidean distance in inverse-dynamics or discriminator representations (ablation) | Episodic intrinsic reward · MiniGrid ObstructedMaze-1Q | exploration efficiency | Helped |
| Use ETD instead of Euclidean distance in inverse-dynamics or discriminator representations (ablation) | Episodic intrinsic reward · MiniGrid DoorKey-16x16 | performance | No clear effect |
| Use minimum temporal distance rather than the 10% quantile, 10th-nearest-neighbor distance, or average of the 1st to 10th nearest neighbors (ablation) | ETD · Ablation environments | exploration efficiency | Helped |
| Replace asymmetric MRN distance with a symmetric distance (ablation) | ETD · Ablation environments | performance | No clear effect |
| Use MRN-POT energy instead of cosine, L2, or MRN energy (ablation) | ETD temporal-distance learning · 17x17 SpiralMaze-Noisy | temporal-distance estimation | Helped |
| Replace symmetric InfoNCE with forward or backward InfoNCE (ablation) | ETD temporal-distance learning · 17x17 SpiralMaze-Noisy | temporal-distance estimation | No clear effect |
| Use cosine energy instead of MRN-POT energy (ablation) | ETD temporal-distance learning · 17x17 SpiralMaze-Noisy | distinguishing distant states | Hurt |
Surprising findings
- Symmetric and asymmetric distance variants of ETD perform nearly identically.
- PPO without intrinsic reward performs well on the MetaWorld and HalfCheetahVelSparse tasks, suggesting they are poor exploration benchmarks.
Limitations
- Purely episodic bonus; does not combine with global bonuses, limiting use in singleton MDPs.
- Episodic bonuses make the task non-Markovian (POMDP-like), possibly biasing value estimates.
- Successor distance may be infinite in non-ergodic settings, assuming ergodic MDPs.
Experimental setup
Datasets: MiniGrid Crafter MiniWorld Maze DeepMind Control MetaWorld HalfCheetahVelSparse
Benchmarks: MiniGrid (MultiRoom, DoorKey, KeyCorridor, ObstructedMaze) MiniGrid with noise Crafter MiniWorld Maze DMC sparse tasks
Metrics: episode return / sample efficiency learning curves Crafter success rate and geometric mean score
Baselines: PPO RND NGU NovelD DEIR E3B EC Count
Builds on: PPO successor distance (CMD) NovelD DEIR EC MRN
Closest papers by meaning
- Revisiting Intrinsic Reward for Exploration in Procedurally Generated Environments (ICLR 2023 · similarity 0.91)
- Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus (NeurIPS 2024 · similarity 0.89)
- Abstract and Explore: A Novel Behavioral Metric with Cyclic Dynamics in Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Episodic Return Decomposition by Difference of Implicitly Assigned Sub-trajectory Reward (AAAI 2024 · similarity 0.86)
- Learning Task Belief Similarity with Latent Dynamics for Meta-Reinforcement Learning (ICLR 2025 · similarity 0.86)
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.