Research map / Reinforcement learning / Policy gradient and actor-critic
ERL-TD: Evolutionary Reinforcement Learning Enhanced with Truncated Variance and Distillation Mutation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ERL-TD, an evolutionary RL method that uses multiple Q-networks with a truncated-variance Bellman backup to curb overestimation and a distillation mutation to guide population mutation, evaluated on OpenAI Gym and DMC.
Problem
Evolutionary reinforcement learning suffers from Q-value overestimation bias in off-policy RL and inefficient, overly random mutations in the EA population, reducing sample efficiency and stability in continuous control.
Approach
Use the mean of multiple Q-networks minus a power of their variance as the Bellman target while variance is unstable (switching to the plain mean once stable), and replace random Gaussian mutation with distillation mutation where population policies distill from the elite/RL teacher and mutate using the last distillation gradient's magnitude and direction.
Key results
- On Ant-v2, ERL-TD nearly doubles the maximum cumulative reward of all baselines and reaches twice the cumulative reward of others at 20k steps on Ant-v2 and Hopper-v2.
- On DMC at 100k steps, ERL-TD (DrQ) reaches 983±14 on reacher-easy versus 722±50 for the best prior baseline shown (SUNRISE), and 972±27 on ball in cup-catch versus 913±53 for DrQ.
- Distillation mutation converges faster and reaches higher discounted reward than Gaussian mutation on Hopper-v2, Swimmer-v2 and Walker2d-v2; ERL-TD with four Q-networks is recommended.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ERL-TD (DrQ) instead of SUNRISE | DMC agent · reacher-easy, 100K steps | score: 722±50 -> 983±14 | Helped |
| Use ERL-TD (DrQ) instead of SUNRISE | DMC agent · ball in cup-catch, 100K steps | score: 913±53 -> 972±27 | Helped |
| Use ERL-TD (DrQ) instead of SUNRISE | DMC agent · walker-walk, 100K steps | score: 667 ±147 -> 475 ±54 | Hurt |
| Use ERL-TD (DrQ) instead of SUNRISE | DMC agent · reacher-easy, 500K steps | score: 982±3 -> 990±5 | Helped |
| Use ERL-TD (DrQ) instead of SUNRISE | DMC agent · walker-walk, 500K steps | score: 953 ±13 -> 889 ±21 | Hurt |
| Use distillation mutation instead of Gaussian mutation (ablation) | ERL-TD · Hopper-v2, Swimmer-v2, and Walker2d-v2 | cumulative reward | Helped |
| Remove variance from the Bellman target (NV) (ablation) | ERL-TD · Hopper-v2 and Walker2d-v2 | cumulative reward | Hurt |
| Use variance without truncation (NT) (ablation) | ERL-TD · Hopper-v2, Swimmer-v2, and Walker2d-v2 | cumulative reward | Hurt |
| Use 2 Q-networks (ablation) | ERL-TD · Swimmer-v2 and Walker2d-v2 | cumulative reward | Hurt |
| Use 3 or 5 Q-networks (ablation) | ERL-TD · Hopper-v2 and Swimmer-v2 | cumulative reward | Hurt |
| Use four Q-networks instead of 2, 3, or 5 (ablation) | ERL-TD · OpenAI Gym environments | cumulative reward | Helped |
| Use REDQ rather than ERL-TD's truncated variance method (ablation) | ERL-TD · Hopper | training time for 10000 frames: 853.74 -> 4,520.63 seconds | Hurt |
Surprising findings
- Running ERL-TD takes about 853.74 s per 10k frames versus 534.52 s for SAC, and the REDQ-based variant takes 4520.63 s, indicating high computational overhead of the variance-based gradient.
- The non-truncated variance variant (NT) performs poorly, especially on Swimmer-v2, showing that truncating the variance penalty is necessary.
Limitations
- Long running time due to computing gradients through multiple Q-networks variance.
- Underperforms or is only close to baselines on several DMC environments at 500k steps.
- Requires a hyperparameter alpha and threshold for truncation, with performance sensitive to alpha choice.
Experimental setup
Datasets: OpenAI Gym (HalfCheetah-v2, Walker2d-v2, Hopper-v2, Ant-v2, Humanoid-v2, Swimmer-v2) DeepMind Control Suite (finger-spin, cartpole-swingup, reacher-easy, cheetah-run, walker-walk, ball in cup-catch)
Benchmarks: OpenAI Gym MuJoCo tasks DeepMind Control Suite
Metrics: cumulative reward mean and standard deviation over seeds Q-value estimation accuracy wall-clock training time
Baselines: ERL ERL-Re2 PDERL CERL CEM-RL SAC TD3 PlaNet Dreamer CURL RAD DrQ
Builds on: ERL CERL PDERL SAC DrQ REDQ
Closest papers by meaning
- Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation (AAAI 2024 · similarity 0.89)
- Value-Evolutionary-Based Reinforcement Learning (ICML 2024 · similarity 0.89)
- EvoRainbow: Combining Improvements in Evolutionary Reinforcement Learning for Policy Search (ICML 2024 · similarity 0.89)
- Evolutionary Reinforcement Learning with Parameterized Action Primitives for Diverse Manipulation Tasks (AAAI 2025 · similarity 0.87)
- Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning (ICLR 2024 · similarity 0.86)
- The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.