Research map / Reinforcement learning / Visual and representation RL
Distributional Successor Features Enable Zero-Shot Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiSPOs, which learn a diffusion-based distribution of successor features of the behavior policy plus a readout policy, enabling zero-shot policy optimization for arbitrary rewards via linear regression and guided sampling without compounding error.
Problem
Transferring RL agents across reward functions in offline settings: model-based RL suffers compounding error over long horizons, while successor features are tied to a specific policy, hindering optimal policy recovery for new tasks.
Approach
Model the distribution of successor features (discounted feature sums) under the behavior policy in the dataset, with a readout policy conditioned on the outcome; at test time, fit reward weights by linear regression and select the best in-distribution outcome via random shooting or guided diffusion.
Key results
- On AntMaze medium-diverse, DiSPO achieves 631 return vs 394 (USFA), 418 (COMBO), 236 (MOPO) and 403 (GC-IQL).
- On Preference Antmaze, DiSPO attains 139 (Up) and 142 (Right) returns while GC-IQL reaches only 72 and 83.
- On Roboverse trajectory stitching, DiSPO achieves 49, 40, and 66 success on PickPlace, ClosedDrawer, BlockedDrawer while RaMP and DT get 0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiSPO rather than the strongest listed baseline | Offline multitask RL · AntMaze and Kitchen tasks in the results table | average episodic return | Mixed |
| Use DiSPO rather than the strongest listed baseline | Offline multitask RL · Hopper tasks | average episodic return | Mixed |
| Use DiSPO rather than COMBO | Preference AntMaze · Up path preference | return: 139 \std{1} vs 143 \std{9} | Hurt |
| Use DiSPO rather than COMBO | Preference AntMaze · Right path preference | return: 136 \std{4} -> 142 \std{2} | Helped |
| Enable outcome-distribution modeling and readout policy to stitch trajectories | DiSPO · Roboverse PickPlace, ClosedDrawer, and BlockedDrawer | success rate: 0 \std{0} -> 49 \std{8}; 0 \std{0} -> 40 \std{5}; 0 \std{0} -> 66 \std{7} | Helped |
| Use guided diffusion instead of random shooting with 1000 particles (ablation) | DiSPO · AntMaze medium-diverse | return: 650 \std{50} vs 631 \std{67} | No clear effect |
| Use guided diffusion instead of random shooting with 1000 particles (ablation) | DiSPO · AntMaze medium-diverse | wall time (s): 94.8 -> 42.9 | Helped |
| Reduce random Fourier feature dimension from 128 to 64 (ablation) | DiSPO · AntMaze medium-diverse | return: 631 \std{67} -> 561 \std{45} | Hurt |
| Reduce random Fourier feature dimension from 128 to 32 or 16 (ablation) | DiSPO · AntMaze medium-diverse | return: 631 \std{67} -> 295 \std{30}; 631 \std{67} -> 307 \std{38} | Hurt |
| Replace random Fourier features with random, forward-dynamics, or Laplacian features (ablation) | DiSPO · AntMaze medium-diverse | return: 631 \std{67} -> 382 \std{43}; 631 \std{67} -> 402 \std{36}; 631 \std{67} -> 376 \std{33} | Hurt |
| Train on randomly subsampled half of the dataset (ablation) | DiSPO · AntMaze medium-diverse | return: 631 \std{67} -> 459 \std{57} | Hurt |
| Remove transitions from the half of the state space containing the test-time goal (ablation) | DiSPO · AntMaze medium-diverse | return: 631 \std{67} -> 390 \std{26} | Hurt |
Surprising findings
- A simple nonparametric nearest-neighbor baseline performs surprisingly well on antmaze-medium-diverse, outperforming RaMP and MOPO.
- Guided diffusion planning is sensitive to the guidance coefficient, requiring per-environment tuning.
Limitations
- Requires features that linearly express rewards; may fail for complex rewards.
- Policy optimality bounded by behavior dataset coverage and skewness.
- Reward inferred only from offline state-reward pairs; online adaptation not fully explored.
Experimental setup
Datasets: D4RL Antmaze D4RL Franka Kitchen Hopper offline dataset Roboverse Preference Antmaze (custom, 1M transitions)
Benchmarks: D4RL AntMaze Franka Kitchen Hopper Preference Antmaze Roboverse
Metrics: average episodic return number of tasks completed (kitchen) wall-clock time
Baselines: USFA FB RaMP MOPO COMBO GC-IQL Decision Transformer Random shooting
Builds on: successor features universal successor features approximators forward-backward representations classifier guidance diffusion DDIM
Closest papers by meaning
- Proto Successor Measure: Representing the Behavior Space of an RL Agent (ICML 2025 · similarity 0.91)
- A Distributional Analogue to the Successor Representation (ICML 2024 · similarity 0.88)
- Zero-Shot Reinforcement Learning from Low Quality Data (NeurIPS 2024 · similarity 0.88)
- Towards Robust Zero-Shot Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.