Research map / Reinforcement learning / Visual and representation RL
Reward-Free Curricula for Training Robust World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WAKER, a reward-free curriculum that samples training environments for world-model learning according to estimated per-environment world-model error (ensemble disagreement), reducing minimax regret across environments.
Problem
Learning a single robust world model from reward-free exploration across many environment instantiations (UPOMDPs), where existing unsupervised environment design methods need task reward signals to select training environments.
Approach
Connects reward-free minimax regret to minimising the maximum latent dynamics (TV) error across environments, and samples environments via a Boltzmann distribution over ensemble-disagreement error estimates, with domain randomisation mixed in.
Key results
- WAKER-M with Plan2Explore improves CVaR_0.1 over domain randomisation across most tasks, e.g. Car Clean Up Sort 0.749 vs 0.549 on OOD extra-block environments and Clean Up Sort 0.973 vs 0.929 average performance.
- WAKER-M achieves statistically significant out-of-distribution gains over DR with both Plan2Explore and random exploration (e.g. Terrain Walker Steep Walk 631.4 vs 512.7 with Plan2Explore).
- Hardest Environment Oracle performs poorly on robustness, while Re-weighting Oracle is strong on Terrain domains but significantly worse than WAKER on Clean Up domains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WAKER-M environment sampling instead of domain randomization | DreamerV2 world-model training with Plan2Explore · OOD environments, including Clean Up Extra Block and Terrain Walker Steep | performance: 0.289 $\pm$ 0.05 -> 0.499 $\pm$ 0.04; 512.7 $\pm$ 31.2 -> 631.4 $\pm$ 20.8 | Helped |
| Use WAKER-M environment sampling instead of domain randomization | DreamerV2 world-model training across Clean Up and Terrain Walker · Robustness CVaR0.1 | performance: 0.089 $\pm$ 0.07 -> 0.348 $\pm$ 0.080; 499.1$\pm$ 43.6 -> 538.5 $\pm$ 42.1 | Helped |
| Prioritize environments with the highest estimated error (WAKER-M) rather than the highest error-reduction rate (WAKER-R) (ablation) | WAKER world-model training · Robustness evaluation across domains | CVaR0.1 performance | Helped |
| Use Plan2Explore rather than random exploration | WAKER and domain randomization · Downstream task performance | performance | Helped |
| Use gradual expansion instead of domain randomization | World-model environment sampling · Robustness evaluation | CVaR0.1 performance | No clear effect |
| Sample only the single most complex environment (HE-Oracle) instead of a range of environments | World-model environment sampling · Robustness evaluation | CVaR0.1 performance | Hurt |
| Focus sampling on the most complex environments using RW-Oracle | World-model environment sampling · Clean Up and Car Clean Up robustness | CVaR0.1 performance | Hurt |
| Use WAKER-M instead of domain randomization | World-model training · Average performance across 100 environments | average performance | Helped |
| Use WAKER-R instead of domain randomization | World-model training · Average performance across 100 environments | average performance | No clear effect |
Surprising findings
- Handcrafted expert curricula (RW-Oracle) are not uniformly better: they are strong on Terrain domains but significantly worse than WAKER on Clean Up.
- Focusing only on the hardest environment (HE-Oracle) yields poor robustness, showing that low error across all environments matters rather than just the most complex one.
Limitations
- Relies on an intrinsically motivated exploration policy to adequately explore across environments, which may not scale to more complex environments.
- Uses a discrete buffer to track uncertainty over environment parameters, which will not scale to large parameter spaces.
- Uncertainty estimates may have differing scales across domains, requiring per-domain normalisation in the multi-domain experiment.
Experimental setup
Datasets: Terrain Walker (DMControl Walker on procedural terrain) Terrain Hopper (DMControl Hopper on procedural terrain) Clean Up (SafetyGym-based) Car Clean Up (SafetyGym-based)
Benchmarks: Terrain Walker Terrain Hopper Clean Up Car Clean Up
Metrics: CVaR_0.1 average performance out-of-distribution performance world model prediction error on worst 10% trajectories probability of improvement (rliable)
Baselines: Domain Randomisation Gradual Expansion Hardest Environment Oracle Re-weighting Oracle Random exploration
Builds on: DreamerV2 Plan2Explore Prioritised Level Replay Unsupervised Environment Design
Closest papers by meaning
- No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery (NeurIPS 2024 · similarity 0.90)
- Inverting the Bellman Equation: From $Q$-Values to World Models (NeurIPS 2026 · similarity 0.88)
- General agents need world models (ICML 2025 · similarity 0.88)
- RLVR-World: Training World Models with Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Learning World Models for Unconstrained Goal Navigation (NeurIPS 2024 · similarity 0.87)
- Marginal Benefit Driven RL Teacher for Unsupervised Environment Design (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.