Research map / Reinforcement learning / Policy gradient and actor-critic
CTD4 – a Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CTD4, a TD3-based actor-critic whose ensemble critics output Gaussian return distributions (mu, sigma) fused via a Kalman-style rule, trained with KL-based distributional TD loss on continuous control tasks.
Problem
Categorical distributional RL requires projection steps, many hyperparameters, and task-specific knowledge, and is largely restricted to discrete actions; overestimation bias in ensembles is typically handled by min or average fusion, which are suboptimal.
Approach
Replace categorical return distributions with continuous Gaussian critics inside a TD3-style actor-critic, so Bellman targets are Gaussian affine transforms and no projection is needed; fuse the ensemble of Gaussian critics with a Kalman fusion rule instead of min or mean.
Key results
- CTD4 is reported to match or outperform TD3 and REDQ across ten DeepMind Control Suite tasks, with outperformance highlighted on Hopper Hop and Acrobot Swingup (5 seeds, 1M steps).
- Kalman fusion is reported to give higher rewards with fewer samples and greater stability than min and average fusion on Cheetah Run and Cartpole Swingup.
- An ensemble size of N=3 critics was selected; N=10 gave suboptimal performance on Reacher Hard and Cartpole Swingup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CTD4 with continuous Gaussian critics and Kalman fusion | TD3 · DMCS continuous-control tasks | evaluation reward | Mixed |
| Replace minimum-value or average fusion with Kalman fusion (ablation) | CTD4 · Cheetah Run and Cartpole Swingup | evaluation reward | Helped |
| Vary the number of critics; use N=3 rather than an excessively large ensemble such as N=10 (ablation) | CTD4 · Reacher Hard and Cartpole Swingup | evaluation reward | Mixed |
| Decay exploration noise over training (ablation) | CTD4 · DMCS continuous-control tasks | evaluation reward | Helped |
Surprising findings
- The authors report that a larger ensemble (N=10) degrades performance, suggesting too many critics can cause instability and underestimation.
Limitations
- Evaluated only on DeepMind Control Suite tasks with a single seed count of five and no statistical tests reported.
- Assumes Gaussian return distributions; exploring other continuous distributions is left to future work.
- Kalman fusion as described is a sequential pairwise fusion whose theoretical justification is loosely stated.
Experimental setup
Datasets: DeepMind Control Suite
Benchmarks: DeepMind Control Suite (ten tasks incl. Reacher Hard, Finger Spin Hard, Ball-in-Cup, Walker Walk, Humanoid Run, Acrobot
Metrics: average evaluation reward learning curve stability across seeds
Baselines: TD3 REDQ
Builds on: TD3 DDPG REDQ Double Q-learning
Closest papers by meaning
- Distributions as Actions: A Unified Framework for Diverse Action Spaces (ICLR 2026 · similarity 0.90)
- Categorical Distributional Reinforcement Learning with Kullback-Leibler Divergence: Convergence and Asymptotics (ICML 2025 · similarity 0.87)
- Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning (ICLR 2025 · similarity 0.86)
- Offline Actor-Critic Reinforcement Learning Scales to Large Models (ICML 2024 · similarity 0.85)
- Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.85)
- Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.