Research map / Learning theory and algorithms / Prediction and decision losses
Quality-Diversity with Limited Resources
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RefQD, which shares one representation network across all decision parts in a quality-diversity archive and uses deep decision archives, periodic top-k re-evaluation and representation learning-rate decay to fix the resulting mismatch, cutting GPU and RAM use to a few percent of PGA-ME/DQN-ME while keeping comparable QD-Score.
Problem
Quality-diversity algorithms need large archives and large populations, so training them is resource-heavy in GPU memory and RAM; prior work targets sample efficiency but not training-time resource efficiency.
Approach
Decompose each policy network into a large shared representation part and small per-solution decision parts, keep only decision parts in the archive, and counter the mismatch between stale decision parts and the updated shared representation with a deep decision archive (K levels), periodic top-k re-evaluation, and learning-rate decay on the representation part.
Key results
- On QDax, RefQD uses 16% GPU memory on Ant Uni versus PGA-ME and matches or exceeds PGA-ME QD-Score on HalfCheetah Uni (104.6%), Humanoid Uni (101.6%), and Ant Trap (107.1%).
- On Atari CNN tasks, RefQD uses 3.7% GPU and 0.3% RAM on Pong and reaches 248.6% of DQN-ME's QD-Score.
- On Atari with ResNet34, DQN-ME runs out of memory while RefQD achieves 220.1% (Pong) and 246.2% (Boxing) of DQN-ME (s) QD-Score.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RefQD instead of PGA-ME | QD algorithms on QDax · HalfCheetah Uni, Humanoid Uni, Ant Trap | QD-Score: 100.0% -> 104.6%; 100.0% -> 101.6%; 100.0% -> 107.1% | Helped |
| Use RefQD instead of PGA-ME | QD algorithms on QDax · Hopper Uni, Walker2D Uni, Ant Uni, Point Maze, Ant Maze | QD-Score: 100.0% -> 92.7%; 100.0% -> 79.6%; 100.0% -> 84.4%; 100.0% -> 85.0%; 100.0% -> 64.9% | Hurt |
| Use RefQD instead of DQN-ME | Atari CNN policies · Pong and Boxing | QD-Score: 100.0% -> 248.6%; 100.0% -> 119.6% | Helped |
| Use RefQD instead of DQN-ME (s) | Atari ResNet policies · Pong and Boxing | QD-Score: 100.0% -> 220.1%; 100.0% -> 246.2% | Helped |
| Reduce offspring population size | PGA-ME on QDax · QDax tasks | QD-Score | Hurt |
| Remove top-k re-evaluation and re-evaluate the whole archive (ablation) | RefQD · QDax | QD-Score | Hurt |
| Remove learning-rate decay for the representation part (ablation) | RefQD · QDax | QD-Score | Hurt |
| Remove DDA and use re-survivor selection (ablation) | Vanilla-RefQD · QDax | QD-Score | Hurt |
| Remove mismatch-mitigation strategies (ablation) | RefQD (Vanilla-RefQD) · QDax | QD-Score | Hurt |
| Increase DDA depth from K=1 to K=4 (ablation) | RefQD · Hopper Uni | QD-Score: 472169 -> 500594 | Helped |
| Increase DDA depth from K=1 to K=4 (ablation) | RefQD · Walker2D Uni, HalfCheetah Uni, Humanoid Uni, Ant Uni, Point Maze | QD-Score: 423790 -> 566845; 2476923 -> 2629326; 587483 -> 646501; 490440 -> 715349; 188707 -> 192868 | Helped |
| Increase DDA depth from K=1 to K=4 (ablation) | RefQD · Ant Maze and Ant Trap | QD-Score: 620880 -> 669850; 919987 -> 741243 | Hurt |
Surprising findings
- RefQD outperforms the unlimited-resource PGA-ME on several QDax tasks despite sharing one representation part.
- Network decomposition is reported to simplify the QD optimization space, yielding better QD-Score than DQN-ME on Atari.
Limitations
- No mathematical theoretical analysis; only empirical evaluation.
- Results shown with a single instantiation based on PGA-ME operators and a fixed decomposition (n+1).
- Top-k re-evaluation and DDA add sample and time cost; some tasks (e.g., Ant Maze, Hopper Uni) show QD-Score drops (64.9%, 92.7%) relative to PGA-ME.
Experimental setup
Datasets: QDax locomotion and maze tasks Atari Pong Atari Boxing
Benchmarks: QDax Hopper Uni QDax Walker2D Uni QDax HalfCheetah Uni QDax Humanoid Uni QDax Ant Uni QDax Point Maze QDax Ant Maze QDax Ant Trap Atari Pong Atari Boxing
Metrics: QD-Score Coverage Max Fitness GPU memory usage RAM usage
Baselines: PGA-ME PGA-ME (s) DQN-ME DQN-ME (s) Vanilla-RefQD EDO-CS
Builds on: PGA-ME MAP-Elites TD3 Double DQN EDO-CS
Closest papers by meaning
- Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization (ICML 2024 · similarity 0.84)
- Quality-Diversity Generative Sampling for Learning with Synthetic Data (AAAI 2024 · similarity 0.82)
- Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learning (ICLR 2024 · similarity 0.82)
- Agent Skill Acquisition for Large Language Models via CycleQD (ICLR 2025 · similarity 0.81)
- Behavioral Foundation Models for Quality Diversity (NeurIPS 2026 · similarity 0.81)
- Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.