Research map / Reinforcement learning / Policy gradient and actor-critic
Distributional value learning
Learn full return distributions with categorical, moment-matching, or spline-based representations.
60 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Policy gradient and actor-critic, keyword cluster “value · td · distributional”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
18 papers in 2023–24 and 15 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 5.2% to 3.2%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Control (ICML 2016 · 56 citations)
- 2017 · A Distributional Perspective on Reinforcement Learning (ICML 2017 · 238 citations)
- 2018 · Distributional Reinforcement Learning With Quantile Regression (AAAI 2018 · 565 citations)
- 2019 · A Comparative Analysis of Expected and Distributional Reinforcement Learning (AAAI 2019 · 42 citations)
- 2020 · Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (ICML 2020 · 50 citations)
- 2021 · Distributional Reinforcement Learning via Moment Matching (AAAI 2021 · 16 citations)
- 2022 · Lyapunov Density Models: Constraining Distribution Shift in Learning-Based Control (ICML 2022 · 5 citations)
- 2023 · Distributional Pareto-Optimal Multi-Objective Reinforcement Learning (NeurIPS 2023 · 4 citations)
- 2024 · Beyond Expected Return: Accounting for Policy Reproducibility When Evaluating Reinforcement Learning Algorithms (AAAI 2024 · 3 citations)
- 2025 · Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning (ICML 2025 · 0 citations)
- 2026 · Path-Coupled Bellman Flows for Distributional Reinforcement Learning (ICML 2026 · 0 citations)
Earliest papers here
- Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Control (ICML 2016 · 56 citations)
- A Distributional Perspective on Reinforcement Learning (ICML 2017 · 238 citations)
- Distributional Reinforcement Learning With Quantile Regression (AAAI 2018 · 565 citations)
What has been tried
47 papers here report 348 measured changes (a component added, removed or swapped, compared with a baseline): helped 60%, no clear effect 11%, hurt 20%, mixed 9%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Optimize a weighted sum of performance and reproducibility with R-ES instead of standard ES | Ant and HalfCheetah, all five noise configurations · MAD | Ant: Action 0.87 -> 0.55; Init-State 0.31 -> 0.11; Obs 0.72 -> 0.15; Param 0.02 -> 0.00; Reward 0.34 -> 0.10. HalfCheeta | Helped | Beyond Expected Return: Accounting for Policy Reproducibility When Evaluating Reinforcement Learning Algorithms (AAAI 2024 · 3 citations) |
| Use only N=2 generated samples | Seaquest · training stability | Hurt | Distributional Reinforcement Learning with Regularized Wasserstein Loss (NeurIPS 2024 · 2 citations) | |
| Increase the ambiguity-set radius to 1.0 | Cliffwalking, p in {0.6, 0.7, 0.8} · cumulative return | Helped | Single-Trajectory Distributionally Robust Reinforcement Learning (ICML 2024 · 1 citation) | |
| Increase the number of features | Tabular MRPs · mean embedding squared error | Helped | Distributional Bellman Operators over Mean Embeddings (ICML 2024 · 1 citation) | |
| Add the DAU advantage shift to DSUP(1/2) | Option trading, risk-neutral (sampled frequencies) · performance | Hurt | Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning (NeurIPS 2024 · 1 citation) | |
| Use a support endpoint larger than the maximum return | Returns supported on [0, 2] · value MSE | Hurt | Categorical Distributional Reinforcement Learning with Kullback-Leibler Divergence: Convergence and Asymptotics (ICML 2025 · 0 citations) | |
| Increase pseudo-return rollout step width k | Examples 1 and 2, on-policy and off-policy · average interval length | Hurt | Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation (NeurIPS 2025 · 0 citations) | |
| Increase ε in the decomposed C51 loss | Most Atari games · learning performance | Helped | Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning (NeurIPS 2025 · 0 citations) | |
| replace diverse ensemble with homogeneous QR/QR ensemble | deep sea · median episodic regret | Hurt | Diverse Projection Ensembles for Distributional Reinforcement Learning (ICLR 2024 · 0 citations) | |
| Use n-step bootstrapping rather than one-step bootstrapping | δ-model training · training stability | Helped | A Distributional Analogue to the Successor Representation (ICML 2024 · 0 citations) | |
| expectile level lowered from 0.3 toward 0.01 (more minimax-like relabeling) | Single-stage and Multi-stage games with optimal adversary · worst-case return | qualitative: smaller alpha yields more robustness | Helped | Adversarially Robust Decision Transformer (NeurIPS 2024 · 0 citations) |
| add QCM variance bonus instead of no variance bonus | CSI 300 · IC | 6.96% -> 7.59% | Helped | AlphaQCM: Alpha Discovery in Finance with Distributional Reinforcement Learning (ICML 2025) |
What reviewers will expect
Datasets and baselines named in the 9 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- QR-DQN · 3 of 9 papers
- C51 · 3 of 9 papers
- IQN · 3 of 9 papers
- DQN · 2 of 9 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 33 papers; from OpenAlex and the papers' affiliations).
- Google DeepMind · 6 papers
- University College London · 4 papers
- Mila · 4 papers
- Google · 3 papers
- University of Alberta · 3 papers
- Shanghai University of Finance and Economics · 2 papers
- Hong Kong University of Science and Technology · 2 papers
- University of Toronto · 2 papers
- Vector Institute · 2 papers
- Harbin Institute of Technology · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Model-based RL with guarantees (245 papers)
- Regularized scalable deep value learning (216 papers)
- Temporal-difference and discounting theory (143 papers)
- Tree search and policy optimization (109 papers)
Elsewhere in the topic
- Safety layers and shielding (85 papers)
- Variance-reduced policy gradients (79 papers)
- Actor-critic theory (76 papers)
- Off-policy actor-critic with expressive policies (71 papers)
- Trajectory reuse in policy gradients (68 papers)
- Importance-weighted OPE estimators (57 papers)
- Alternative policy update rules (50 papers)
- Constrained policy optimization (48 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).