Research map / Reinforcement learning / Policy gradient and actor-critic
Double Buffers CEM-TD3: More Efficient Evolution and Richer Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DBCEM-TD3, which improves CEM-TD3 with an actor buffer (more efficient CEM evolution via fitness decay) and a critic buffer (independent TD3 critics for more diverse actors), evaluated on MuJoCo tasks.
Problem
CEM-TD3 has low policy search efficiency because CEM discards half of its actors each iteration, and TD3 gradient guidance with a shared critic reduces population diversity.
Approach
Maintain an actor buffer of size K that stores elite actors across iterations, so only N=3 new actors are generated per iteration and replace the worst buffer actors; decay buffer fitness exponentially to avoid premature convergence. Also keep a critic buffer of N independent critics, each trained on the shared replay buffer and matched randomly with new actors, to diversify TD3 guidance.
Key results
- On Ant-v2 at 1M timesteps, DBCEM-TD3 reaches 5483.31 return vs 4381.43 for CEM-TD3, a 25.09% final performance gain.
- DBCEM-TD3 improves sample efficiency by 55.61% over CEM-TD3 on Ant-v2 and raises population diversity from 0.12 to 0.41.
- DBCEM-TD3 has a 12.50% time overhead relative to CEM-TD3 (207 vs 184 minutes on Ant-v2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DBCEM-TD3 instead of CEM-TD3 | CEM-TD3 · Ant-v2 | average return: 4381.43±395.45 -> 5483.31±296.61 | Helped |
| Use DBCEM-TD3 instead of CEM-TD3 | CEM-TD3 · HalfCheetah-v2 | average return: 10503.32±806.29 -> 11174.98±422.71 | Helped |
| Use DBCEM-TD3 instead of CEM-TD3 | CEM-TD3 · Hopper-v2 | average return: 3667.23±70.09 -> 3698.08±35.48 | Helped |
| Use DBCEM-TD3 instead of CEM-TD3 | CEM-TD3 · Swimmer-v2 | average return: 89.75±36.24 -> 97.17±40.52 | Mixed |
| Use DBCEM-TD3 instead of CEM-TD3 | CEM-TD3 · Walker2d-v2 | average return: 4381.43±395.45 -> 5483.31±296.61 | Helped |
| Use DBCEM-TD3 instead of CEM-TD3 | CEM-TD3 · Ant-v2 | population diversity: 0.12 -> 0.41 | Helped |
| Generate N=3 new actors per iteration and retain actors in an actor buffer instead of sampling K=10 each evolution | DBCEM-TD3 · Ant-v2 | evolutionary number: 137 -> 397 | Helped |
| Set new actors per iteration to N=3 (ablation) | DBCEM-TD3 · HalfCheetah-v2 | average return | Helped |
| Set fitness decay factor to α=0.7 (ablation) | DBCEM-TD3 · HalfCheetah-v2 | average return | Helped |
| Use independently trained critics in a critic buffer to guide newly generated actors (ablation) | CEM-TD3 · Ant-v2 | population diversity | Helped |
Surprising findings
- On Swimmer-v2, evolutionary methods (ERL, PDERL, CERL) outperform DBCEM-TD3 and CEM-TD3, which the authors attribute to large reward fluctuation.
Limitations
- Evaluated only on five MuJoCo tasks from OpenAI Gym (v2 environments).
- Swimmer-v2 results show RL-focused methods, including DBCEM-TD3, fail to find high-return policies where evolutionary methods succeed.
- Hyperparameters (N=3, alpha=0.7) were tuned on HalfCheetah-v2 and then reused across environments.
Experimental setup
Datasets: MuJoCo continuous control tasks (Ant-v2, HalfCheetah-v2, Hopper-v2, Swimmer-v2, Walker2d-v2)
Benchmarks: Ant-v2 HalfCheetah-v2 Hopper-v2 Swimmer-v2 Walker2d-v2
Metrics: average return sample efficiency evolutionary efficiency population diversity time cost
Baselines: CEM-TD3 TD3 SAC ERL PDERL CERL
Builds on: CEM-TD3 TD3 CEM ERL Bootstrapped DQN
Closest papers by meaning
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.84)
- Diffusion Actor-Critic with Entropy Regulator (NeurIPS 2024 · similarity 0.84)
- Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation (AAAI 2024 · similarity 0.84)
- Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (NeurIPS 2026 · similarity 0.84)
- ACE: Off-Policy Actor-Critic with Causality-Aware Entropy Regularization (ICML 2024 · similarity 0.84)
- Learning mirror maps in policy mirror descent (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.