Research map / Reinforcement learning / Visual and representation RL

Evolutionary Reinforcement Learning with Parameterized Action Primitives for Diverse Manipulation Tasks

AAAI 2025 · Reinforcement learning › Visual and representation RL · approach: Skill learning and diffusion policies · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ERLAP, which combines hierarchical RL over parameterized action primitives (SAC-trained) with a dual-population evolutionary module that stores and evolves successful primitive sequences to improve exploration and avoid catastrophic forgetting in manipulation tasks.

Problem

Robotic manipulation tasks require long sequences of actions; RL suffers from inefficient exploration over huge action spaces and catastrophic forgetting of successful experiences under sparse rewards.

Approach

Integrates a library of parameterized action primitives (HRL, high-level type policy plus low-level parameter networks trained with SAC) with an evolutionary module that treats whole primitive sequences as individuals, using a success population and a UCB-sampled development population plus crossover and four mutation operators to diversify the replay buffer.

Key results

• Under dense rewards, ERLAP reaches 99.27% average success rate across six Robosuite tasks versus 98.55% for TRAPs, 97.25% for MAPLE, and 12.5%/21% for TD3/SAC.
• ERLAP uses 18.8% fewer steps than TRAPs and 30.1% fewer than MAPLE on dense-reward tasks.
• Under sparse rewards, ERLAP achieves 77.6% on Put-into-Drawer where MAPLE and TRAPs get 0%, and 81.8% on Stack versus 74.6% for MAPLE+PER.

Measured effects reported in the paper

Surprising findings

• Plain TD3 and SAC obtain near-zero success on complex tasks such as Pick-and-Place, Nut-Assembly, Peg-Insert and Clean-Up, showing the value of primitives for exploration.

Limitations

• Primitive library is predefined by hand rather than learned automatically.
• Success on sparse-reward tasks is still modest (e.g., 77.6% on Put-into-Drawer, 81.8% on Stack).
• Only 3 random seeds were used for reported curves.

Experimental setup

Datasets: Robosuite simulated manipulation tasks Unity sparse-reward manipulation tasks

Benchmarks: Robosuite Lift Robosuite Pick-and-Place Robosuite Nut-Assembly Robosuite Peg-Insert Robosuite Clean-Up Robosuite Stack Unity Lift Unity Pick-and-Place Unity Stack Unity Put-into-Drawer

Metrics: task success rate average reward curves sampling steps experience proportion in replay buffer

Baselines: TD3 SAC MAPLE TRAPs MAPLE+PER

Builds on: SAC Khadka and Tumer ERL MAPLE TRAPs PER

Closest papers by meaning

• ERL-TD: Evolutionary Reinforcement Learning Enhanced with Truncated Variance and Distillation Mutation (AAAI 2024 · similarity 0.87)
• PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learning (ICLR 2025 · similarity 0.86)
• TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning (ICLR 2025 · similarity 0.86)
• EvoRainbow: Combining Improvements in Evolutionary Reinforcement Learning for Policy Search (ICML 2024 · similarity 0.85)
• Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation (AAAI 2024 · similarity 0.85)
• Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.85)