Research map / Reinforcement learning / Policy gradient and actor-critic
Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QDAC, an off-policy actor-critic that uses a value critic and a successor-features critic under a Lagrangian constraint to learn high-performing, diverse skill-conditioned policies, with a model-based variant QDAC-MB built on DreamerV3.
Problem
Learning a single skill-conditioned policy that executes a large set of diverse skills (defined by expected features) while maximizing return, in continuous control locomotion tasks.
Approach
Combine a value function critic (performance) and a successor-features critic (skill execution) in a constrained actor objective; a Lagrange multiplier per state and skill is adapted via a cross-entropy loss to switch between maximizing return and satisfying the skill constraint, using a bound that relates expected features to successor features.
Key results
- QDAC achieves 15% more diverse behaviors and 38% higher performance than other baselines across six continuous control tasks (abstract/introduction claim).
- QDAC and QDAC-MB outperform all baselines in executing a wide range of skills, except Humanoid Jump where DCG-ME and QD-PG have a better distance profile for the model-free variant.
- On the hierarchical Ant-Wall task QDAC's learned skills achieve significantly higher performance than other baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QDAC's value and successor-features critics with constrained optimization | QDAC · Six continuous-control locomotion tasks | performance: 38% higher performance | Helped |
| Use QDAC's value and successor-features critics with constrained optimization | QDAC · Six continuous-control locomotion tasks | behavior diversity: 15% more diverse behaviors | Helped |
| Use QDAC-MB's world-model-based variant | QDAC-MB · Humanoid Jump | distance to skill | Helped |
| Use QDAC rather than PPGA, DCG-ME, or QD-PG | QDAC · Velocity-feature locomotion tasks | distance to skill | Helped |
| Use QDAC-MB rather than Fixed-λ | QDAC-MB · All evaluated skill-learning tasks | distance to skill | Helped |
| Use a successor-features critic rather than naive distance-to-skill minimization (ablation) | QDAC-MB · All evaluated skill-learning tasks | distance to skill | Helped |
| Use QDAC rather than No-SF or UVFA (ablation) | QDAC · All evaluated skill-learning tasks | distance to skill | Helped |
| Use adaptive λ rather than a fixed quality-diversity trade-off (ablation) | QDAC-MB · All evaluated skill-learning tasks | skill coverage | Helped |
| Use adaptive λ rather than a fixed quality-diversity trade-off (ablation) | QDAC-MB · Humanoid Jump | performance | Helped |
| Use QDAC rather than DCG-ME or QD-PG | QDAC · Humanoid Jump | distance to skill | Hurt |
| Use QDAC rather than PPGA | QDAC · Humanoid motor failure | return | Hurt |
| Use QDAC skills for adaptation rather than other baselines | QDAC · Humanoid motor failure (100% damage) | return: more than 5,000 | Helped |
Surprising findings
- No-SF and UVFA can only execute skills at corners of the skill space on feet contact tasks, as they minimize naive feature distance.
- QDAC does not seem able to adapt to gravity variations, while the model-based variant does competitively.
Limitations
- Relies on a manually defined feature function for diversity; the authors note most QD methods share this limitation.
- Model-free QDAC performs poorly on gravity perturbations and Humanoid Jump relative to some baselines.
- Only evaluated on locomotion tasks; non-ergodic environments such as Atari are left to future work.
Experimental setup
Datasets: Brax simulated locomotion environments (Walker, Ant, Humanoid)
Benchmarks: Brax Walker/Ant/Humanoid with feet contact, velocity, jump, angle features few-shot adaptation: Humanoid Hurdles, Humanoid Motor Failure, Ant Gravity, Walker Friction Ant Wall hierarchical task
Metrics: distance score performance score distance profiles performance profiles probability of improvement
Baselines: PPGA DCG-ME QD-PG DOMiNO SMERL Reverse SMERL No-SF Fixed-lambda UVFA
Builds on: SAC DreamerV3 successor features UVFA DCG-ME DOMiNO
Closest papers by meaning
- Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization (ICML 2024 · similarity 0.88)
- Actor-Free Continuous Control via Structurally Maximizable Q-Functions (NeurIPS 2025 · similarity 0.87)
- Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.87)
- Behavioral Foundation Models for Quality Diversity (NeurIPS 2026 · similarity 0.87)
- Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.