Research map / Continual learning / Online and replay-based
Learning Successor Features the Simple Way
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Simple SF, a pixel-based method that learns successor features online with only a TD loss on Q-values plus a stop-gradient reward-prediction loss on L2-normalized features, avoiding representation collapse without pretraining or auxiliary losses.
Problem
Learning successor features from pixel observations in non-stationary deep RL suffers from representation collapse, while existing fixes rely on reconstruction, orthogonality, pretraining, or complex multi-loss schemes that are costly or slow.
Approach
Use L2-normalized encoder outputs as basis features, train the task vector w only with a stop-gradient reward prediction loss, and train SFs with a Q-SF-TD loss on the bootstrapped target; the reward-prediction term keeps features non-constant, so collapse to a constant is not a minimizer.
Key results
- Simple SF matches or outperforms reconstruction, orthogonality, random-feature and APS baselines across 2D Minigrid, 3D Four Rooms and Mujoco continual RL settings, with 5 seeds.
- Simple SF reaches the performance threshold in fewer steps and trains in less total time than orthogonality-constrained SF baselines in the slippery Four Rooms efficiency analysis.
- Canonical SF learning shows representation collapse, with average cosine similarity between SFs converging towards 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Simple SF instead of baseline agents | Continual RL with pixel observations · 2D Minigrid | cumulative return | Helped |
| Use Simple SF instead of baseline agents | Continual RL with pixel observations · 3D Four Rooms | cumulative return | Helped |
| Use Simple SF instead of baseline agents | Continual RL with pixel observations · Mujoco | normalized return | Helped |
| Add a stop-gradient operator to basis features in the reward prediction loss (ablation) | Simple SF · Four Rooms | learning performance | Helped |
| Use Simple SF instead of other SF methods | Continual RL with stochastic dynamics · Slippery Four Rooms, second task | learning performance | Helped |
| Add reconstruction constraints to basis features | SF learning agents · Continual RL | cumulative return | Hurt |
| Add orthogonality constraints to basis features | SF learning agents · Continual RL | cumulative return | Hurt |
| Use Simple SF instead of SFs with reconstruction constraints or random basis features | SF representations · Center-Wall | SR decoding error | Mixed |
| Remove the stop-gradient operator from the reward prediction loss (ablation) | Simple SF · Four Rooms | learning performance | Hurt |
Surprising findings
- Constraints such as reconstruction and orthogonality on basis features can impede RL learning even when they produce well-clustered representations.
- Removing the stop-gradient on basis features in the reward prediction loss causes learning instability and representational collapse (phi driven toward zero).
Limitations
- Evaluated predominantly in simulated environments, not real-world settings.
- Assumes good-quality pixel observations; poor image quality could degrade performance.
- Mujoco experiments use a single cycle of two tasks, a narrower continual evaluation.
Experimental setup
Datasets: 2D Minigrid Miniworld Four Rooms Mujoco
Benchmarks: 2D Minigrid Center-Wall 2D Minigrid Inverted-LWalls 3D Miniworld Four Rooms Slippery Four Rooms Mujoco half-cheetah/walker continual
Metrics: cumulative return episode return steps to threshold frames per second training duration silhouette score Davies-Bouldin score decoding MSE
Baselines: Double DQN reconstruction-constrained SF orthogonality-constrained SF random basis features SF APS (pre-train) SF + Q-TD + Reward
Builds on: Universal Successor Features Double DQN APS URL Benchmark
Closest papers by meaning
- Successor Feature Sets: Generalizing Successor Representations Across Policies (AAAI 2021 · similarity 0.91)
- Reward-Aware Proto-Representations in Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Distributional Successor Features Enable Zero-Shot Policy Optimization (NeurIPS 2024 · similarity 0.86)
- Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making (ICML 2024 · similarity 0.85)
- Bridging State and History Representations: Understanding Self-Predictive RL (ICLR 2024 · similarity 0.85)
- A Distributional Analogue to the Successor Representation (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.