Research map / Reinforcement learning / Policy gradient and actor-critic
Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TERL, a two-stage evolutionary RL framework where a population of full actor-critic TD3 agents shares a replay buffer and PSO updates, first exploring independently and then concentrating compute on the best individual.
Problem
Existing evolutionary RL methods use only actor networks as EA individuals, which limits exploration because the whole actor population stalls when the shared critic falls into local optima; maintaining full RL agents per individual is computationally costly.
Approach
Maintain a population of complete actor-critic agents that share a common replay buffer and PSO-based actor updates, with a two-stage schedule: an exploration stage where all individuals train independently and an exploitation stage where only the best individual receives RL gradient updates while others continue PSO-driven optimization.
Key results
- TERL achieves 12632 on HalfCheetah and 6149 on Walker2d, outperforming TD3 (11880, 5386) and the other baselines in most environments.
- TERL is slightly below ERL-Re2 on Ant (6611 vs 7210) and LunarLander (277 vs 283), ranking second in those environments.
- The best individual in the exploration stage approaches or exceeds a single TD3 agent using only about 20% of interaction and training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TERL instead of TD3 | TD3 · HalfCheetah | episode return: 11880 ± 1327 -> 12632 ± 400 | Helped |
| Use TERL instead of ERL | ERL · Swimmer | episode return: 358 ± 4 -> 366 ± 2 | Helped |
| Use TERL instead of TD3 | TD3 · Reacher | episode return: -3.7 ± 0.5 -> -3.7 ± 0.4 | No clear effect |
| Use TERL instead of PDERL | PDERL · Hopper | episode return: 3546 ± 97 -> 3736 ± 24 | Helped |
| Use TERL instead of ERL-Re2 | ERL-Re2 · Ant | episode return: 7210 ± 93 -> 6611 ± 291 | Hurt |
| Use TERL instead of ERL-Re2 | ERL-Re2 · Walker2d | episode return: 5807 ± 582 -> 6149 ± 219 | Helped |
| Use TERL instead of ERL-Re2 | ERL-Re2 · BipedalWalker | episode return: 304 ± 11 -> 311 ± 8 | Helped |
| Use TERL instead of ERL-Re2 | ERL-Re2 · LunarLander | episode return: 283 ± 4 -> 277 ± 5 | Hurt |
| Use TERL instead of TD3 | TD3 · Pendulum | episode return: −153 ± 24 -> -148 ± 18 | Helped |
| remove the exploration stage (ablation) | TERL · Hopper, Ant, and Walker2d | episode return | Hurt |
| remove the exploitation stage (ablation) | TERL · Hopper, Ant, and Walker2d | episode return | Hurt |
| Share experiences through a common replay buffer and PSO updates | TERL individuals in the exploration stage · Hopper, Ant, and Walker2d | episode return | Helped |
Surprising findings
- The TERL variant without the exploitation stage performs nearly as well as the full method, suggesting the extra evaluation and updates to the best individual achieve a similar effect to the exploitation stage.
Limitations
- Performance is weaker early in training because multiple agents are trained concurrently.
- Slightly inferior to ERL-Re2 on Ant and LunarLander.
Experimental setup
Datasets: MuJoCo OpenAI Gym classic control (BipedalWalker, LunarLander, Pendulum)
Benchmarks: HalfCheetah Swimmer Reacher Hopper Ant Walker2d BipedalWalker LunarLander Pendulum
Metrics: episodic return (sum of rewards) mean and standard deviation over five seeds
Baselines: ERL PDERL ERL-Re2 TD3 PPO
Builds on: ERL CERL CEM-RL PDERL TD3
Closest papers by meaning
- ERL-TD: Evolutionary Reinforcement Learning Enhanced with Truncated Variance and Distillation Mutation (AAAI 2024 · similarity 0.89)
- Value-Evolutionary-Based Reinforcement Learning (ICML 2024 · similarity 0.87)
- EvoRainbow: Combining Improvements in Evolutionary Reinforcement Learning for Policy Search (ICML 2024 · similarity 0.87)
- Structured Reinforcement Learning for Combinatorial Decision-Making (NeurIPS 2025 · similarity 0.85)
- Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.85)
- Evolutionary Reinforcement Learning with Parameterized Action Primitives for Diverse Manipulation Tasks (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.