Research map / Reinforcement learning / Policy gradient and actor-critic
Diverse Projection Ensembles for Distributional Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes projection ensembles combining quantile and categorical distributional RL models, with an ensemble-disagreement (1-Wasserstein) bonus propagated through distributional TD errors for deep exploration (PE-DQN).
Problem
Exploration and epistemic uncertainty estimation in distributional RL: projections onto parametric return-distribution representations impose inductive biases that shape ensemble generalization, but existing distributional ensembles lack reliable epistemic uncertainty estimates for exploration.
Approach
Form an equal mixture of distributional models using different projections and representations (quantile and categorical), prove contraction of the projection mixture operator, and use average pairwise 1-Wasserstein ensemble disagreement, propagated via a temporal-consistency bonus iteration, as an optimism bonus for directed exploration.
Key results
- On deep sea grids up to size 100, PE-DQN roughly halves median episodic regret relative to baselines.
- On bsuite, PE-DQN improves most strongly on the exploration capability while using 4 distributional models versus 20 DQN models in BDQN+P and IDS ensembles.
- On VizDoom MyWayHome, only PE-DQN finds the goal across all 10 seeds in all three sparsity variations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use diverse quantile and categorical projections instead of a single projection (ablation) | PE-DQN · Deep sea | episodic regret | Helped |
| Train ensemble members independently and select actions directly using ensemble disagreement, without uncertainty propagation (ablation) | PE-DQN · Deep sea | episodic regret | Hurt |
| Use PE-DQN's distributional projection ensemble and uncertainty propagation | PE-DQN · bsuite | core capability scores | Helped |
| Use PE-DQN instead of the tested baselines | PE-DQN · Deep sea, sizes up to 100 | median episodic regret | Helped |
| Use PE-DQN instead of the tested algorithms | PE-DQN · VizDoom MyWayHome variations | successful object finding across seeds | Helped |
| Choose a projection that shapes predictive-distribution statistics for exploration | Categorical and quantile distributional models · Deep sea, size 50 | exploration | Mixed |
Surprising findings
- The sparse VizDoom variant was harder for baselines, which the authors attribute to the map's forkedness.
Limitations
- Only uniform mixtures over ensemble members were considered.
- Projection errors create residual epistemic uncertainty that does not vanish at convergence.
- Hyperparameters were tuned by grid/Optuna search on subsets of problems.
Experimental setup
Datasets: bsuite VizDoom MyWayHome
Benchmarks: bsuite VizDoom MyWayHome deep sea
Metrics: bsuite core capability score episodic regret success rate across seeds
Baselines: BDQN+P IDS-C51 DLTV QR-DQN
Builds on: QR-DQN C51 Bootstrapped DQN with prior functions Information-directed sampling DLTV
Closest papers by meaning
- Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- The Curse of Diversity in Ensemble-Based Exploration (ICLR 2024 · similarity 0.88)
- Improving Generalization in Offline Reinforcement Learning via Latent Distribution Representation Learning (AAAI 2025 · similarity 0.87)
- Categorical Distributional Reinforcement Learning with Kullback-Leibler Divergence: Convergence and Asymptotics (ICML 2025 · similarity 0.86)
- Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model (NeurIPS 2024 · similarity 0.86)
- Foundations of Multivariate Distributional Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.