Research map / Reinforcement learning / Visual and representation RL
Unlocking the Power of Representations in Long-term Novelty-based Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RECODE, a non-parametric clustering-based online density estimator for novelty-based exploration that tracks visitation counts over thousands of episodes, paired with a masked-transformer representation (CASM) and evaluated on Atari hard-exploration games and DM-HARD-8.
Problem
Novelty-based exploration in deep RL needs visitation-count estimates that generalize over large state spaces, remain responsive to recent experience, and are robust to noisy or uncontrollable observation features; existing non-parametric memories reset every episode and parametric bonuses like RND
Approach
Replace per-episode slot memories with a shared, fixed-size set of cluster atoms updated online via a DP-means-inspired rule with count discounting and probabilistic removal of under-populated clusters, yielding a soft visitation count; additionally a masked-transformer multi-step action-prediction embedding (CASM) generalizes the inverse dynamics loss.
Key results
- RECODE sets state-of-the-art on the hard-exploration Atari subset and is the first agent to reach the end screen of Pitfall!.
- On DM-HARD-8 (single-task), RECODE solves six of eight tasks with super-human performance and is the first to beat the human baseline on Push Blocks.
- Under noisy-observation Montezuma's Revenge, RECODE's performance is not significantly degraded while MEME-NGU-AP deteriorates significantly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace NGU with RECODE while keeping the MEME agent and AP representation | MEME exploration on Atari hard-exploration games · Atari hard-exploration subset | game score | Helped |
| Use RECODE with CASM representations | MEME exploration · DM-HARD-8 | capped human-normalized score | Helped |
| Replace AP with CASM (ablation) | NGU · DM-HARD-8 | capped human-normalized score | Helped |
| Use CASM instead of AP (ablation) | NGU and RECODE · DM-HARD-8 | aggregated performance | Helped |
| Replace NGU with RECODE while using AP representations | MEME exploration · DM-HARD-8 | capped human-normalized score | Helped |
| Replace NGU with RECODE while using CASM representations | MEME exploration · DM-HARD-8 | capped human-normalized score | Helped |
| Add noisy observation frames (ablation) | MEME-NGU-AP · Noisy Montezuma's Revenge | game score | Hurt |
| Add noisy observation frames (ablation) | MEME-RECODE-AP · Noisy Montezuma's Revenge | game score | No clear effect |
| Remove cluster-count discounting (set γ=1) (ablation) | RECODE with concurrently trained embeddings · Atari hard-exploration games | game score | Hurt |
| Remove cluster-count discounting (set γ=1) (ablation) | RECODE with pretrained frozen embeddings · Atari | game score | No clear effect |
| Change the atom-removal strategy from probability proportional to 1/c² to another tested strategy (ablation) | RECODE · Atari | game score | Mixed |
| Use episodic memory alone (EMM) instead of RECODE (ablation) | MEME with AP representations · Montezuma's Revenge | game score | Hurt |
Surprising findings
- Discounting cluster counts is necessary when representations are trained concurrently; with gamma=1 performance degrades markedly on Montezuma's Revenge and Pitfall!.
- A variant with no discount still performs reasonably when the representation is frozen from a pretrained snapshot.
Limitations
- Requires careful tuning of memory size, discount, insertion probability and kernel parameters across domains.
- Higher variance in performance under noisy observations compared with clean setting.
Experimental setup
Datasets: Atari hard-exploration games (Pitfall!, Montezuma's Revenge, etc.) DM-HARD-8
Benchmarks: Atari hard-exploration subset (8 games) DM-HARD-8 Noisy Montezuma's Revenge
Metrics: game score human-normalized score capped human-normalized score
Baselines: NGU RND BYOL-Explore EMM MEME-NGU-AP
Builds on: Never Give Up (NGU) MEME DP-means BYOL-Explore RND
Closest papers by meaning
- BYOL-Explore: Exploration by Bootstrapped Prediction (NeurIPS 2022 · similarity 0.88)
- Just Cluster It: An Approach for Exploration in High-Dimensions using Clustering and Pre-Trained Representations (ICML 2024 · similarity 0.88)
- Count-Based Exploration with Neural Density Models (ICML 2017 · similarity 0.87)
- Novelty Search in Representational Space for Sample Efficient Exploration (NeurIPS 2020 · similarity 0.87)
- Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Models (ICLR 2025 · similarity 0.85)
- Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.