Research map / Reinforcement learning / Visual and representation RL
EvoRainbow: Combining Improvements in Evolutionary Reinforcement Learning for Policy Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically compares mechanisms of evolutionary-RL hybrids across five perspectives on dense, deceptive, and sparse reward tasks, and combines the best ones into EvoRainbow and EvoRainbow-Exp.
Problem
Existing evolutionary reinforcement learning (ERL) methods propose many mechanisms for combining EAs and RL, but it is unclear which are complementary and can be fully combined, and prior evaluations focus mainly on dense-reward MuJoCo tasks.
Approach
Decouple ERL algorithms into five perspectives (interaction mode, individual architecture, EA and operators, EA impact on RL, fitness surrogate usage), evaluate each mechanism step by step on dense, deceptive, and sparse reward tasks, then fuse the best per perspective into two frameworks: EvoRainbow for general tasks and EvoRainbow-Exp for exploration tasks.
Key results
- EvoRainbow and EvoRainbow-Exp outperform existing ERL baselines and standard RL on 20 tasks spanning MinAtar, MuJoCo, maze, trap, and manipulation tasks.
- Parallel interaction mode outperforms EA-Master and RL-Master modes on maze and trap tasks, where only the parallel mode escapes the local optimum in Trap Maze.
- CEM is found to outperform V-GA, PD-GA, and B-GA in five of six tasks, and Genetic Soft Update generally beats Elite Guide.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Parallel mode instead of EA-Master or RL-Master mode | EvoRainbow framework · Locomotion, maze, and trap tasks | average episode return | Helped |
| Use Shared Architecture instead of Private Architecture | Parallel-mode ERL · Humanoid and Ant | average episode return | Helped |
| Use Private Architecture instead of Shared Architecture | Parallel-mode ERL · Trap Maze, Wall Task, and Large Maze (Sparse) | average episode return | Helped |
| Replace V-GA with CEM | ERL with Shared or Private Architecture · Five of six evaluated tasks | average episode return | Helped |
| Replace CEM with PD-GA | ERL · Trap Maze | average episode return | Hurt |
| Add Genetic Soft Update from elite individuals to the RL policy | PSC and PPC · Evaluated tasks | average episode return | Mixed |
| Add Elite Policy Guide | PSC and PPC · Maze and trap tasks | average episode return | Hurt |
| Use Critic rather than PeVFA as the fitness estimator | ERL fitness surrogate · Ant and Trap Maze | Spearman rank correlation | Helped |
| Use Critic (H-step Bootstrap) with Generation Control | PSCG and PPCG · Locomotion tasks | average episode return | Helped |
| Use Critic (H-step Bootstrap) with Generation Control | PSCG and PPCG · Trap Maze and Wall Task | average episode return | Hurt |
| Combine Parallel mode, Shared Architecture, CEM, Genetic Soft Update, and Critic (H-step Bootstrap) with Generation Control | EvoRainbow · General tasks | average episode return | Helped |
| Use Private Architecture, CEM, and Genetic Soft Update without the fitness surrogate | EvoRainbow-Exp · Exploration tasks | average episode return | Helped |
Surprising findings
- Shared architecture performs better on locomotion but worse than private architecture on sparse and deceptive reward tasks, because value-function-based shared representations degrade with low-quality rewards.
- PeVFA-based surrogate fitness has lower ranking correlation and higher approximation error than Critic, and fitness surrogates do not help on exploration tasks.
Limitations
- Mechanisms were decoupled and compared step by step, so not all combinations of mechanisms were examined and interactions may be missed.
- No single framework covers all task types; users must choose between EvoRainbow and EvoRainbow-Exp.
Experimental setup
Datasets: MuJoCo MinAtar Maze tasks (PointMaze/AntMaze) Trap tasks Robot manipulation tasks (Meta-World)
Benchmarks: MuJoCo Humanoid MuJoCo Ant MuJoCo Walker MuJoCo Swimmer MinAtar Ant Maze Trap Maze Wall Task Large Maze (Sparse) Robot manipulation tasks
Metrics: average episodic return success rate Spearman rank correlation
Baselines: ERL CERL PDERL CEM-RL PGPS ERL-Re2 SAC TD3 DQN
Builds on: ERL CEM-RL PDERL ERL-Re2 PGPS CERL TD3 SAC
Closest papers by meaning
- ERL-TD: Evolutionary Reinforcement Learning Enhanced with Truncated Variance and Distillation Mutation (AAAI 2024 · similarity 0.89)
- Value-Evolutionary-Based Reinforcement Learning (ICML 2024 · similarity 0.88)
- Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation (AAAI 2024 · similarity 0.87)
- Helix: Evolutionary Reinforcement Learning for Open-Ended Scientific Problem Solving (ICLR 2026 · similarity 0.86)
- ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learning (AAAI 2025 · similarity 0.85)
- Evolutionary Reinforcement Learning with Parameterized Action Primitives for Diverse Manipulation Tasks (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.