Research map / Reinforcement learning / Policy gradient and actor-critic

Value-Evolutionary-Based Reinforcement Learning

ICML 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Temporal-difference and discounting theory

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VEB-RL, which evolves a population of Q-networks using negative TD error as fitness and lets only elite individuals interact with the environment, improving DQN, Rainbow, and SPR on MinAtar and Atari.

Problem

Combining evolutionary algorithms with RL has mostly targeted policy-based RL with cumulative-reward fitness, leaving value-based RL underexplored; uniform interaction of all population members also injects low-quality samples and wastes samples.

Approach

Maintain a population of Q-networks (with target networks) evolved by GA or CEM, ranked by negative TD error computed from replay buffer samples without environment interaction, and use Elite Interaction so only top-N individuals generate experience for the shared replay buffer while the RL agent is injected into the population each generation.

Key results

• On MinAtar, VEB-RL yields about 100% improvement over DQN on Asterix and Seaquest and about 50% on Breakout and Space Invaders.
• GA-based VEB-SPR improves SPR by roughly 40% on Atari 100k, outperforming baselines on seven of nine tasks (e.g., Battle Zone 15505 vs 9190 for SPR).
• On MuJoCo, GA-based VEB-TD3 reaches Ant 7356 vs TD3 6219 and Walker 5490 vs 4806 (average returns).

Measured effects reported in the paper

Surprising findings

• Using VEB-RL with N=0 (no elite interaction) still slightly outperforms DQN, but performance declines as the number of interacting elites N increases.
• Performance collapses when RL is prohibited from interacting with the environment, attributed to distributional shift.

Limitations

• No theoretical guarantees; supported only by experiments.
• Adds computational and time overhead from maintaining a population of Q-networks and target networks.
• Fitness uses random sampling from the replay buffer, which may not ensure state coverage.

Experimental setup

Datasets: MinAtar Atari Atari 100k MuJoCo

Benchmarks: MinAtar Atari Atari 100k MuJoCo

Metrics: undiscounted return Spearman rank correlation TD error elite rate

Baselines: ERL CEM-RL ERL-Re2 PDERL CURL OTRainbow SimPLe SPR TD3 DQN Rainbow

Builds on: ERL CEM-RL DQN Rainbow SPR

Closest papers by meaning

• Evolution-Guided Policy Gradient in Reinforcement Learning (NeurIPS 2018 · similarity 0.89)
• ERL-TD: Evolutionary Reinforcement Learning Enhanced with Truncated Variance and Distillation Mutation (AAAI 2024 · similarity 0.89)
• EvoRainbow: Combining Improvements in Evolutionary Reinforcement Learning for Policy Search (ICML 2024 · similarity 0.88)
• Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation (AAAI 2024 · similarity 0.87)
• A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.85)
• Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.85)