Research map / Reinforcement learning / MDPs and planning
REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes REValueD, which adds an ensemble of critics and a regularisation loss to DecQN value-decomposition for factorisable MDPs, showing reduced target variance and better performance on discretised DM Control tasks.
Problem
Discrete-action RL with high-dimensional factorisable action spaces suffers from the combinatorial explosion of atomic actions; value-decomposition (DecQN) reduces this but its effect on target bias/variance and credit assignment across sub-actions is unclear.
Approach
Theoretically shows DecQN decomposition lowers target over-estimation bias but raises target variance; counters this with an ensemble of critics (variance scales as 1/K), and adds a Huber-based regularisation loss on individual sub-action utilities to limit the effect of exploratory sub-actions on optimal sub-action values.
Key results
- REValueD outperforms DecQN and BDQ across most DM Control tasks, with especially large gains on humanoid and dog tasks (N=21 and 38 sub-action spaces).
- On humanoid and dog tasks, DecQN 604.82 vs DecQN+Ensemble 832.99 vs REValueD 915.80 on Humanoid-Stand; Dog-Run 625.68 vs 750.73 vs 821.17.
- Theoretically, DecQN lowers expected target difference but increases its variance; an ensemble of K critics reduces variance by factor 1/K while leaving the mean unchanged.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add an ensemble of critics to DecQN (ablation) | DecQN · Humanoid-Stand | return: 604.82 ± 85.1 -> 832.99 ± 11.3 | Helped |
| Add the regularisation loss to DecQN+Ensemble (ablation) | DecQN+Ensemble · Humanoid-Stand | return: 832.99 ± 11.3 -> 915.80 ± 6.12 | Helped |
| Add the regularisation loss to DecQN+Ensemble (ablation) | DecQN+Ensemble · Humanoid-Walk | return: 817.63 ± 7.66 -> 874.33 ± 3.63 | Helped |
| Add the regularisation loss to DecQN+Ensemble (ablation) | DecQN+Ensemble · Humanoid-Run | return: 478.11 ± 4.59 -> 534.81 ± 10.8 | Helped |
| Add the regularisation loss to DecQN+Ensemble (ablation) | DecQN+Ensemble · Dog-Walk | return: 819.95 ± 22.0 -> 862.31 ± 12.0 | Helped |
| Add the regularisation loss to DecQN+Ensemble (ablation) | DecQN+Ensemble · Dog-Trot | return: 878.47 ± 6.33 -> 902.01 ± 7.65 | Helped |
| Add the regularisation loss to DecQN+Ensemble (ablation) | DecQN+Ensemble · Dog-Run | return: 750.73 ± 11.2 -> 821.17 ± 8.10 | Helped |
| Increase ensemble size from 1 to 15 (ablation) | REValueD · Dog-Run | return: 675.51 ± 9.10 -> 834.77 ± 4.16 | Helped |
| Increase ensemble size from 1 to 3 (ablation) | REValueD · Walker-Run | return: 763.97 ± 4.75 -> 779.40 ± 1.29 | Helped |
| Set regularisation coefficient β to 0 instead of 0.5 (ablation) | REValueD · Dog-Walk | return: 886.73 ± 7.38 -> 838.81 ± 14.3 | Hurt |
| Vary ensemble size (ablation) | REValueD · Walker-Run, Cheetah-Run, Quadruped-Run, and dog tasks | return | Mixed |
Surprising findings
- BDQ fails completely to learn in humanoid environments, and performs consistently on dog-walk where DecQN falls short at large n_i.
- The sum value-decomposition (DecQN-Sum) generally performs worse than the mean decomposition, consistent with its theoretically higher overestimation bias.
Limitations
- Variance analysis relies on an assumed i.i.d. uniform noise model for approximation error.
- Ensemble size gives no consistent best value; smaller ensembles can be better on easier tasks.
- Regularisation weight beta was not tuned and gains on some tasks (e.g. finger-spin with beta=0) are mixed.
Experimental setup
Datasets: DeepMind Control Suite (discretised, Bang-Off-Bang) MetaWorld (appendix)
Benchmarks: DeepMind Control Suite (humanoid, dog, walker, cheetah, quadruped, fish, finger) MetaWorld
Metrics: episode return asymptotic return (mean ± standard error) CVaR of detrended gradient norms
Baselines: DecQN BDQ DecQN-Sum DecQN+Ensemble
Builds on: DecQN Double Q-learning Ape-X Branching Dueling Q-Networks
Closest papers by meaning
- Decoupling regularization from the action space (ICLR 2024 · similarity 0.88)
- Supported Value Regularization for Offline Reinforcement Learning (NeurIPS 2023 · similarity 0.88)
- Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML 2024 · similarity 0.87)
- Demonstration-Regularized RL (ICLR 2024 · similarity 0.86)
- Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.