Research map / Reinforcement learning / Policy gradient and actor-critic

SAPG: Split and Aggregate Policy Gradients

ICML 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Trajectory reuse in policy gradients · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SAPG, an on-policy RL method that splits parallel environments among multiple latent-conditioned policies and aggregates their data into a leader policy via clipped importance-sampled off-policy PPO updates, scaling to ~24k environments.

Problem

Standard on-policy RL such as PPO saturates in performance when scaled to tens of thousands of GPU-simulated parallel environments, because i.i.d. Gaussian action sampling yields many near-duplicate transitions and wasted environment capacity.

Approach

Divide environments into blocks, each run by a separate policy sharing a backbone with block-specific latent parameters (with entropy regularization on followers for diversity); a leader policy is updated with PPO on its own data plus importance-weighted off-policy data from the followers, while followers update only on their own data.

Key results

• SAPG reaches 35.7 successes on Regrasping vs 31.9 for PBT and 1.25 for PPO, and 23.7 on Throw vs 19.2 for PBT and 16.8 for PPO.
• On Two Arms Reorientation, SAPG (entropy 0.005) gets 28.58 successes vs 14.46 for PBT and 1.73 for PPO, more than twice PBT.
• On AllegroHand, SAPG (entropy 0) reaches 1.23e4 reward vs 1.01e4 for PPO and PQL, a roughly 21% margin over PQL.

Measured effects reported in the paper

Surprising findings

• Vanilla PPO and PQL make essentially no progress on the hard AllegroKuka tasks even at 24k environments.
• Symmetric aggregation, where all policies learn from all data, performs significantly worse, suggesting policies converge in behavior and diversity is lost.

Limitations

• Performance on easier tasks (ShadowHand) is comparable to PQL rather than clearly better, and PQL is more sample-efficient early.
• Method is evaluated only on dexterous manipulation tasks in IsaacGym; no evaluation on other domains is reported.
• Hyperparameters such as the entropy coefficient and number of policies (M=6) are tuned per environment from a small set.

Experimental setup

Datasets: IsaacGym simulated Allegro-Kuka, Allegro Hand and Shadow Hand tasks

Benchmarks: AllegroKuka Regrasping AllegroKuka Throw AllegroKuka Reorientation AllegroKuka Two Arms Reorientation AllegroHand in-hand reorientation ShadowHand in-hand reorientation

Metrics: number of successes per episode episode reward

Baselines: PPO PBT (DexPBT) PQL (Parallel Q-Learning)

Builds on: PPO Population-Based Training (DexPBT) Parallel Q-Learning IsaacGym

Closest papers by meaning

• Accelerated Policy Gradient: On the Convergence Rates of the Nesterov Momentum for Reinforcement Learning (ICML 2024 · similarity 0.87)
• Gradient Boosting Reinforcement Learning (ICML 2025 · similarity 0.86)
• Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
• Convergence Analysis of Policy Gradient Methods with Dynamic Stochasticity (ICML 2025 · similarity 0.86)
• Policy Gradient with Tree Expansion (ICML 2025 · similarity 0.86)
• Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees (ICLR 2024 · similarity 0.86)