Research map / Reinforcement learning / Visual and representation RL
Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIVA, a quality-diversity (MAP-Elites/CMA-ES) approach that generates diverse meta-RL training tasks in open-ended simulators using a small set of downstream feature samples, outperforming UED baselines (ACCEL, PLR⊥) with fewer environment steps.
Problem
Training adaptive meta-RL agents needs diverse training tasks, but hand-designed or randomized generators in complex open-ended simulators produce little meaningful diversity, and UED methods waste compute evaluating redundant levels.
Approach
Use quality-diversity optimization over an unstructured genotype space, with an archive whose feature axes are defined by domain knowledge and whose bounds/sampling are guided by a few target-distribution feature samples, then meta-train over tasks drawn from the archive density.
Key results
- On Racing, DIVA outperforms all baselines, including UED methods given 3x more environment interactions.
- DIVA transfers zero-shot to human-designed F1 tracks and significantly outperforms the ODS oracle trained directly on the downstream distribution.
- On Alchemy, PLR⊥ and ACCEL fail to generate stone sets with phenotypic diversity needed for downstream success, while DIVA matches the target distribution.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DIVA's QD updates to target goal-location diversity | GridNav agents trained with VariBAD · GridNav | episodic return | Helped |
| Use DIVA's QD updates to target goal-location diversity | GridNav agents trained with VariBAD · GridNav | success rate | Helped |
| Use DIVA to generate diverse training stone states | Alchemy agents trained with VariBAD · Alchemy downstream distribution | performance | Helped |
| Use DIVA's QD updates to generate diverse training tracks | Racing agents trained with VariBAD · Racing downstream distribution | performance | Helped |
| Train with DIVA-generated tasks | Racing agents trained with VariBAD · F1 tracks | track completion rate | Helped |
| Add PLR-perp-based level selection to DIVA on the misspecified archive | DIVA+ · Racing | performance | Helped |
| Add PLR-perp-based level selection to DIVA on the main archive | DIVA+ · Racing | performance | No clear effect |
| Add sample-mask updates during stage 1 (ablation) | DIVA · Alchemy | target-region archive coverage | Helped |
| Vary the QD mutation rate (ablation) | DIVA · Alchemy | results | No clear effect |
| Use more QD updates (ablation) | DIVA · Alchemy | performance | Helped |
| Use fewer QD updates (10k in each stage) (ablation) | DIVA · Alchemy | performance | Hurt |
| Use fewer downstream feature samples to estimate the target distribution (ablation) | DIVA · Alchemy | performance | Mixed |
Surprising findings
- DIVA outperforms the ODS oracle trained directly on the downstream F1-track distribution, which the authors attribute to sharper angles in generated tracks.
- Combining DIVA with PLR⊥ (DIVA+) improves a slightly misspecified archive but does not significantly improve the main archive.
Limitations
- Requires access to downstream feature samples and pre-specified axes of diversity.
- Archive design (features, bounds, resolution) is heuristic and hand-tuned for some domains.
Experimental setup
Datasets: GridNav Alchemy (symbolic) Racing Formula-1 tracks
Benchmarks: GridNav Alchemy Racing Formula-1 tracks transfer
Metrics: episodic return success rate track completion rate goal diversity
Baselines: ODS DR PLR⊥ ACCEL VariBAD
Builds on: VariBAD ACCEL PLR MAP-Elites pyribs
Closest papers by meaning
- Human-Timescale Adaptation in an Open-Ended Task Space (ICML 2023 · similarity 0.90)
- Meta-Reinforcement Learning with Zero-Shot Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- Skill-aware Mutual Information Optimisation for Zero-shot Generalisation in Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers (NeurIPS 2024 · similarity 0.88)
- Unsupervised Training Sequence Design: Efficient and Generalizable Agent Training (AAAI 2024 · similarity 0.88)
- Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.