Research map / Reinforcement learning / Policy gradient and actor-critic
A Distributional Analogue to the Successor Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the distributional successor measure (SM), a reward-agnostic distribution over state occupancy measures learned as a delta-model ensemble of generative models trained with a two-level MMD loss, enabling zero-shot distributional policy evaluation.
Problem
Distributional RL requires rewards at training time, so return distributions for new reward functions must be relearned; the successor representation only gives zero-shot expected values, and model-based rollouts accumulate error.
Approach
Define the distributional successor measure as the distribution over random discounted state-occupancy measures, which factorises return distributions into a reward-agnostic object and any deterministic reward; learn it via a delta-model (equally weighted generative model atoms) with a two-level MMD loss, n-step bootstrapping, and adversarial adaptive kernels.
Key results
- On Pendulum held-out rewards, the distributional SM substantially outperforms Rollout, gamma-model ensemble and Mean baselines in Cramer distance to Monte Carlo return distributions.
- The distributional SM correctly ranks two policies by both mean and 0.4-CVaR for held-out reward functions, which the SM-based zero-shot methods cannot do for CVaR.
- Learning the distributional SM is shown to avoid the accumulation of model error seen in rollouts from a learned transition model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the δ-model distributional successor measure for zero-shot return-distribution prediction instead of learned-transition rollouts | δ-model · Pendulum held-out reward functions | Cramér distance | Helped |
| Use the δ-model distributional successor measure for zero-shot return-distribution prediction instead of a γ-model ensemble | δ-model · Pendulum held-out reward functions | Cramér distance | Helped |
| Use the δ-model distributional successor measure for zero-shot return-distribution prediction instead of a Dirac mass at the Monte Carlo expected return | δ-model · Pendulum held-out reward functions | Cramér distance | Helped |
| Use n-step bootstrapping rather than one-step bootstrapping (ablation) | δ-model · δ-model training | training stability | Helped |
| Adapt the model-kernel bandwidth using the median heuristic (ablation) | δ-model · δ-model training | training | Helped |
| Use an adaptive adversarial state kernel instead of a non-adaptive state kernel (ablation) | δ-model · Pendulum | Wasserstein distance | Mixed |
| Vary the number of model atoms (ablation) | δ-model · Pendulum | Wasserstein distance | Mixed |
Surprising findings
- Rollouts from a learned transition model fail to produce reasonable return distributions due to error accumulation over long trajectories.
- The gamma-model ensemble estimates return distributions about as well as a Dirac at the ground-truth mean, despite not modelling aleatoric uncertainty.
Limitations
- Reward functions must be deterministic; stochastic rewards break the linear factorisation except in cycle-less environments.
- Evaluation is limited to two simple domains (Windy Gridworld and Pendulum).
Experimental setup
Datasets: Windy Gridworld (simulated) Pendulum (simulated)
Benchmarks: Windy Gridworld Pendulum
Metrics: Cramer distance Wasserstein distance CVaR at level 0.4 mean return
Baselines: ensemble of gamma-models learned transition kernel rollouts (Rollout) Mean (Dirac at MC expected return)
Builds on: successor representation gamma-models distributional reinforcement learning MMD-GAN iResNet
Closest papers by meaning
- Successor Feature Sets: Generalizing Successor Representations Across Policies (AAAI 2021 · similarity 0.90)
- Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model (NeurIPS 2024 · similarity 0.89)
- Distributional Successor Features Enable Zero-Shot Policy Optimization (NeurIPS 2024 · similarity 0.88)
- Reward-Aware Proto-Representations in Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Proto Successor Measure: Representing the Behavior Space of an RL Agent (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.