Research map / Reinforcement learning / Visual and representation RL
DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that value-loss level prioritisation reduces mutual information between agent representation and training levels, and proposes DRED, which augments training levels with a VAE trained on an initial level set to improve zero-shot generalisation in Minigrid.
Problem
Zero-shot generalisation of deep RL agents across environment levels; understanding why adaptive level sampling reduces the generalisation gap, and how to generate additional training levels without causing distributional shift away from the target context distribution.
Approach
Adaptive value-loss level prioritisation is interpreted as rejection sampling that minimises mutual information between the shared representation and level identity (bounding the generalisation gap). DRED then augments a limited training set with levels generated by a VAE pre-trained on that set, mixing sampling so generated levels stay grounded to the target distribution.
Key results
- VAE-DRED reaches about 1.2x the returns of the next best baseline on held-out Minigrid levels, with statistically significant gains over other methods.
- On Procgen, value-loss prioritisation (S=S^V) reaches a normalised test score of 130.3% versus 100% for uniform sampling, and average level-classifier accuracy falls from 49% to 35%.
- VAE-DRED solves about three times as many Hardcore levels as the next best baseline and achieves over three times DR's IQM on in-context edge cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use value-loss prioritised level sampling instead of uniform sampling | PPO · Procgen test levels | normalised test score: 100.0 ± 1.4 -> 130.3 ± 5.3 | Helped |
| Prioritise levels with low mutual information instead of value-loss sampling | PPO · Procgen test levels | normalised test score: 130.3 ± 5.3 -> 97.0 ± 4.0 | Hurt |
| Mix MI scoring with value-loss scoring | PPO · Procgen test levels | normalised test score: 130.3 ± 5.3 -> 131.3 ± 8.4 | No clear effect |
| Add uniform sampling to value-loss prioritisation | PPO · Procgen test levels | normalised test score: 130.3 ± 5.3 -> 125.4 ± 5.3 | No clear effect |
| Replace the VAE level generator with ACCEL local edits (ablation) | VAE-DRED · Minigrid X_test | mean score: 0.82 ± 0.02 -> 0.77 ± 0.05 | Hurt |
| Replace the VAE level generator with ACCEL local edits (ablation) | VAE-DRED · Minigrid X_train | mean score: 0.92 ± 0.00 -> 0.90 ± 0.03 | No clear effect |
| Set the DRED mixing coefficient η to 1 throughout training (ablation) | EL-DRED · Minigrid | performance | Hurt |
| Generate additional levels with VAE-DRED rather than DR | PPO · Minigrid in-context edge cases | IQM: over three times DR's IQM | Helped |
| Generate additional levels with VAE-DRED rather than DR | PPO · Minigrid in-context edge cases | solved rate: twice its solved rate | Helped |
| Generate additional levels with VAE-DRED rather than DR | PPO · Minigrid in-context Hardcore levels | solved levels: three times as many levels | Helped |
| Use value-loss prioritisation instead of uniform sampling | PPO · Procgen games | average mutual information: 3.31 ± 0.03 -> 2.61 ± 0.05 | Helped |
| Use value-loss prioritisation instead of uniform sampling | PPO · Procgen games | test score: 130.3 ± 5.3 | Mixed |
Surprising findings
- Minimising mutual information too aggressively (S=S^MI) degrades training and final test performance, since the MI bound governs the generalisation gap rather than test returns.
- Existing UED methods such as ACCEL and RPLR cause large distributional shift and perform poorly on test levels despite a near-zero generalisation gap.
Limitations
- Experiments are limited to a gridworld (Minigrid) and Procgen, with no evaluation in more complex environments.
- The generalisation bound is loose and the connection between MI and test returns is empirical rather than tight.
- The approach relies on a pre-collected set of level parameters X_train that is assumed to come from the target distribution.
Experimental setup
Datasets: Procgen Minigrid procedurally generated levels (512 X_train)
Benchmarks: Procgen Minigrid held-out test levels (X_test, 2048) Minigrid in-context edge cases (448) Minigrid Hardcore levels (216)
Metrics: mean episodic return IQM solved rate generalisation gap shift-induced gap mutual information estimate classifier accuracy normalised test score
Baselines: Uniform sampling PLR Domain randomisation RPLR ACCEL ACCEL-D EL-DRED MI-scoring prioritisation
Builds on: Prioritised Level Replay ACCEL Robust PLR Domain randomisation PPO
Closest papers by meaning
- Improving Regret Approximation for Unsupervised Dynamic Environment Generation (NeurIPS 2025 · similarity 0.88)
- Adversarial Environment Design via Regret-Guided Diffusion Models (NeurIPS 2024 · similarity 0.88)
- How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity (NeurIPS 2024 · similarity 0.87)
- Zero-Shot Reinforcement Learning from Low Quality Data (NeurIPS 2024 · similarity 0.87)
- Zero Shot Generalization of Vision-Based RL Without Data Augmentation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.