Research map / Reinforcement learning / Policy gradient and actor-critic
A Comparative Analysis of Expected and Distributional Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CDF-gradient distributional updates instead of expected-value Q-learning updates | Tabular Q-learning · 12x12 gridworld | performance | No clear effect |
| Use PMF-gradient updates instead of CDF-gradient updates | Tabular distributional Q-learning · 12x12 gridworld | performance | Mixed |
| Use PMF-gradient updates instead of CDF-gradient updates | Tabular distributional Q-learning · 3-state MDP | performance | Hurt |
| Use CDF-gradient updates instead of expected-value updates | S51-lite · CartPole, Fourier order 4 | learning speed | Hurt |
| Use PMF-gradient updates instead of CDF-gradient updates | S51-lite · CartPole, Fourier order 4 | performance | Hurt |
| Use distributional methods rather than DQN-lite with a low-order Fourier basis | Linear value-function approximation · CartPole and Acrobot, Fourier basis orders 1–3 | performance | Hurt |
| Replace Fourier features with a two-layer ReLU network | C51 and DQN · CartPole | performance | Helped |
| Use C51's softmax output and cross-entropy loss instead of S51's linear probability output and Cramér loss | Distributional RL · Acrobot, nonlinear two-layer ReLU network | performance | Helped |
| Use a nonlinear function approximator rather than a linear function approximator | Distributional RL · Control tasks | performance | Mixed |
Closest papers by meaning
- Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model (NeurIPS 2024 · similarity 0.90)
- Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning (ICML 2024 · similarity 0.89)
- Categorical Distributional Reinforcement Learning with Kullback-Leibler Divergence: Convergence and Asymptotics (ICML 2025 · similarity 0.89)
- Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.