Research map / Reinforcement learning / Visual and representation RL
Proto Successor Measure: Representing the Behavior Space of an RL Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Proto Successor Measure (PSM), a zero-shot RL method that learns a policy-independent affine basis for all successor measures from reward-free data, so optimal policies for any reward are obtained by solving a small linear program over basis weights.
Problem
Zero-shot reinforcement learning: producing optimal policies for arbitrary downstream reward functions in a fixed environment without additional interaction, using only reward-free data.
Approach
Shows that all successor measures satisfying the Bellman flow constraints form an affine set, so they can be written as a policy-independent basis Phi times policy-specific weights w plus a bias b; a single-player objective with a discrete codebook of policy seeds learns Phi and b, and at test time a reward is optimized by a constrained linear program over w.
Key results
- On gridworld, PSM policy error is 2.05 vs 19.28 (Laplace) and 14.53 (FB); on discrete maze 11.54 vs 38.47 and 28.80.
- PSM performs consistently better or competitive to Laplacian, FB and HILP across DM Control tasks at representation dimension 128.
- PSM's zero-shot performance is stable across latent dimension d from 32 to 256 on Walker and Cheetah, unlike HILP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PSM instead of Laplacian | zero-shot policy prediction · Gridworld | average policy error: 19.28 ± 2.34 -> 2.05 ± 1.20 | Helped |
| Use PSM instead of FB | zero-shot policy prediction · Gridworld | average policy error: 14.53 ± 0.68 -> 2.05 ± 1.20 | Helped |
| Use PSM instead of Laplacian | zero-shot policy prediction · Discrete Maze | average policy error: 38.47 ± 7.01 -> 11.54 ± 1.07 | Helped |
| Use PSM instead of FB | zero-shot policy prediction · Discrete Maze | average policy error: 28.80 ± 10.50 -> 11.54 ± 1.07 | Helped |
| Use affine-space dimension d=32 instead of d=128 (ablation) | PSM · Walker (average across tasks) | return: 689.07 -> 678.27 | No clear effect |
| Use affine-space dimension d=50 instead of d=128 (ablation) | PSM · Walker (average across tasks) | return: 689.07 -> 669.45 | No clear effect |
| Use affine-space dimension d=256 instead of d=128 (ablation) | PSM · Walker (average across tasks) | return: 689.07 -> 690.86 | No clear effect |
| Use affine-space dimension d=32 instead of d=128 (ablation) | PSM · Cheetah (average across tasks) | return: 607.61 -> 628.79 | No clear effect |
| Use affine-space dimension d=50 instead of d=128 (ablation) | PSM · Cheetah (average across tasks) | return: 607.61 -> 615.61 | No clear effect |
| Use affine-space dimension d=256 instead of d=128 (ablation) | PSM · Cheetah (average across tasks) | return: 607.61 -> 615.64 | No clear effect |
Surprising findings
- Forward-backward performance drops during training on Fetch-Reach, which the authors attribute to overestimation from Bellman optimality backups.
Limitations
- Unclear how to choose the size of the representation dimension d.
- Representation quality may depend on dataset coverage, which is not studied.
- Theoretical claim that value-function spans are a subset of successor-measure spans holds only under a specific construction.
Experimental setup
Datasets: ExoRL Fetch-Reach exploratory data (DQN+RND) gridworld/four-room reward-free transitions
Benchmarks: Gridworld Four-room Discrete Maze Fetch-Reach DM Control Walker DM Control Cheetah DM Control Quadruped DM Control Pointmass
Metrics: average policy error vs oracle zero-shot task return
Baselines: Laplacian features Forward-Backward HILP
Builds on: successor features forward-backward representations proto value functions HILP ExoRL
Closest papers by meaning
- Distributional Successor Features Enable Zero-Shot Policy Optimization (NeurIPS 2024 · similarity 0.91)
- Reward-Aware Proto-Representations in Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Zero-Shot Reinforcement Learning from Low Quality Data (NeurIPS 2024 · similarity 0.89)
- Zero-Shot Reinforcement Learning via Function Encoders (ICML 2024 · similarity 0.88)
- A Distributional Analogue to the Successor Representation (ICML 2024 · similarity 0.87)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.