Research map / Reinforcement learning / Visual and representation RL
Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts on-policy PPO into a differentiable quality diversity framework (CMA-MAEGA-based) with vectorized PPO, Markovian measure proxies, xNES, and PPO-based walking of the search policy, achieving the first QD-RL solution of the Humanoid locomotion task.
Problem
Existing quality diversity reinforcement learning methods rely on off-policy RL or evolution strategies and scale poorly to high-dimensional, stochastic locomotion tasks such as Humanoid, leaving a performance gap to standard RL.
Approach
Uses PPO as the objective and measure gradient estimator inside a DQD framework (CMA-MAEGA), with a vectorized PPO that jointly optimizes objective and measure proxies, Markovian measure proxies for foot-contact measures, xNES in place of CMA-ES for stability, and walking the search policy by PPO on a gradient-coefficient-weighted reward.
Key results
- On Humanoid, PPGA achieves roughly a 4x improvement in best reward and QD-score over baselines, and is the only algorithm that solves the Humanoid domain (best reward 9677 in the ablation table).
- PPGA outperforms baselines in best reward and QD-score on most tasks and achieves comparable coverage except on Ant; the results are averaged over four seeds with 95% bootstrapped confidence intervals.
- Post-hoc re-evaluation (50 evaluations per agent) shows PPGA keeps the best-reward lead on all tasks and the QD-score lead on Humanoid and Ant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove measure functions (ablation) | PPGA · Humanoid | Best Reward: 9677 -> 9653 | No clear effect |
| add measure functions (ablation) | PPGA · Humanoid | Coverage: 32.06% -> 98.67% | Helped |
| replace PPGA with standard PPO and archive intermediate policies (ablation) | PPGA · Humanoid | Best Reward: 9677 -> 9651 | No clear effect |
| replace PPGA with standard PPO and archive intermediate policies (ablation) | PPGA · Humanoid | Coverage: 98.67% -> 0.08% | Hurt |
| use (5, 10) instead of the baseline (10, 10) for (N1, N2) (ablation) | PPGA · Humanoid | performance | No clear effect |
| use (10, 5) instead of the baseline (10, 10) for (N1, N2) (ablation) | PPGA · Humanoid | coverage | Hurt |
| reduce (N1, N2) to (1, 1) from (10, 10) (ablation) | PPGA · Humanoid | performance | Hurt |
| replace PPO with TD3 (ablation) | PPGA · Humanoid | best reward | Hurt |
| walk the search policy with VPPO instead of weighted gradient recombination (ablation) | PPGA · Humanoid | performance | Helped |
| replace xNES with CMA-ES (ablation) | PPGA · RL environments | training stability | Hurt |
Surprising findings
- TD3 used as a drop-in replacement for PPO (TD3GA) fails to find high-performing policies on Humanoid, with xNES step size exploding to reach 100% coverage.
- Reducing PPO steps (N1=1, N2=1) drops performance to slightly over 50% of baseline, while the PPO + Archive ablation reaches the same best reward but under 1% coverage.
Limitations
- On-policy PPO is sample-inefficient and requires many parallel environments per agent.
- Enabling PPO's adaptive action standard deviation hurts exploration, as seen in lower coverage on Ant.
- PPGA underperforms baselines in coverage on Ant and relies on Brax simulation with measures limited to foot contact.
Experimental setup
Datasets: Brax Ant Brax Walker2d Brax Half-Cheetah Brax Humanoid
Benchmarks: Brax Ant Brax Walker2d Brax Half-Cheetah Brax Humanoid
Metrics: QD-score coverage best reward CCDF corrected QD-score corrected coverage corrected best reward
Baselines: PGA-ME QDPG CMA-MAEGA (TD3, ES) sep-CMA-MAE TD3GA PBT-ME (SAC) PPO + Archive PPGA (No Measures)
Builds on: CMA-MAEGA PPO CMA-MEGA xNES QDax
Closest papers by meaning
- Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization (ICML 2024 · similarity 0.89)
- Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics (ICML 2024 · similarity 0.88)
- Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.87)
- DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization (AAAI 2024 · similarity 0.87)
- Behavioral Foundation Models for Quality Diversity (NeurIPS 2026 · similarity 0.87)
- Learning Diverse Risk Preferences in Population-Based Self-Play (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.