Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Imagined Autocurricula
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a diffusion world model on a mixed offline Procgen dataset and then trains an actor-critic agent purely inside imagined rollouts, using Prioritized Level Replay over imagined initial states and random imagination horizons (iMac) to generalize to unseen levels.
Problem
Generalizing to unseen procedurally generated levels from a limited offline dataset is hard for offline RL and model-free methods; generating useful imagined training data from world models is also inefficient without proper curriculum or horizon choice.
Approach
Combine a history-conditioned EDM diffusion world model with reward/termination ensemble predictors and PLR-based prioritization over imagined initial states and randomly sampled imagination horizons, inducing an automatic curriculum over imagined worlds without controlling environment generation.
Key results
- iMac achieves mean test returns of 6.20 on CoinRun, 5.78 on Jumper, and 5.41 on Maze, outperforming the best model-free baseline by 17%, 48%, and 35% respectively.
- Compared with a fixed 15-step imagination horizon world model, iMac improves Jumper by up to 56% and Maze by 38%.
- Ablation on Jumper: random horizons improve over fixed horizon by 11% (4.11 vs 3.71) and PLR prioritization adds a further 41% (5.78).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with iMac imagined autocurriculum instead of the best model-free baseline | Offline RL on Procgen · CoinRun held-out levels | mean return: 17% | Helped |
| Train with iMac imagined autocurriculum instead of the best model-free baseline | Offline RL on Procgen · Ninja held-out levels | mean return: 19% | Helped |
| Train with iMac imagined autocurriculum instead of the best model-free baseline | Offline RL on Procgen · Jumper held-out levels | mean return: 48% | Helped |
| Train with iMac imagined autocurriculum instead of the best model-free baseline | Offline RL on Procgen · Maze held-out levels | mean return: 35% | Helped |
| Train with iMac imagined autocurriculum instead of the best model-free baseline | Offline RL on Procgen · CaveFlyer held-out levels | mean return: 24% | Helped |
| Train with iMac imagined autocurriculum instead of the best model-free baseline | Offline RL on Procgen · Heist held-out levels | mean return: 38% | Helped |
| Train with iMac imagined autocurriculum instead of the best model-free baseline | Offline RL on Procgen · Miner held-out levels | mean return: 19% | Helped |
| Add random imagined horizons instead of fixed 15-step horizons (ablation) | World model agent training · Jumper held-out levels | mean return: 3.71 -> 4.11; 11% | Helped |
| Add PLR-based prioritization to random-horizon imagined rollouts (ablation) | World model agent training · Jumper held-out levels | mean return: 4.11 -> 5.78; 41% | Helped |
Surprising findings
- Model-free offline RL methods such as CQL and IQL perform worse than behavior cloning on several Procgen games in this setup.
Limitations
- Computational requirements are substantial and effectiveness varies across environment types.
- Evaluated only on Procgen 64x64 visual environments with discrete actions; continuous control not tested.
- Reward signals are sparse and the world model must learn long-horizon dynamics without intermediate reward.
Experimental setup
Datasets: Procgen mixed offline dataset (expert/medium/random, 200 levels)
Benchmarks: Procgen (CoinRun, Ninja, Jumper, Maze, CaveFlyer, Heist, Miner) held-out levels
Metrics: mean return on held-out test levels
Baselines: BC CQL IQL BCQ BCT Decision Transformer WM Fixed WM Random
Builds on: DIAMOND Prioritized Level Replay Elucidating Diffusion Models Advantage Actor-Critic Unsupervised Environment Design
Closest papers by meaning
- Learning Interactive Real-World Simulators (ICLR 2024 · similarity 0.88)
- Exploration-Driven Generative Interactive Environments (CVPR 2025 · similarity 0.87)
- SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning (NeurIPS 2026 · similarity 0.87)
- Genie: Generative Interactive Environments (ICML 2024 · similarity 0.86)
- Reward-Free Curricula for Training Robust World Models (ICLR 2024 · similarity 0.86)
- AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.