Research map / Reinforcement learning / Visual and representation RL
Unsupervised Training Sequence Design: Efficient and Generalizable Agent Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Unsupervised Training Sequence Design (UTSD), a finite-horizon teacher MDP that encodes student validation performance and trains a meta-teacher with PEARL to design short efficient curricula for students.
Problem
Unsupervised environment design (UED) methods need very large numbers of training environments and are inefficient, so they are unsuitable when environment creation is costly or training efficiency is critical; the paper targets efficient finite-horizon training sequences that also transfer to unsee
Approach
Formulate the teacher as a finite-horizon MDP whose state encodes the student's validation performance over Quality-Diversity-selected validation environments, rewarded by that performance, and train it as a context-based meta-RL agent (PEARL) across diverse students so it can adapt to new students quickly.
Key results
- Meta-teacher converges to training sequences of average length 9.08 ± 0.13 in Bit-Flipping, versus 12 for the human-designed sequence, while UED teachers do not shorten the sequence.
- In Lunar-Lander, meta-teacher reduces the training sequence length from 12 to 10.38 ± 0.11, whereas UED methods fail to find sequences shorter than 12.
- In Minigrid, the meta-teacher consistently produces the best aggregate IQM and optimality gap across five runs among compared UED teachers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the meta-teacher to design the training sequence instead of the human-designed sequence | Bit-Flipping student training · Bit-Flipping | training sequence length: 12 -> 9.08 ± 0.13 | Helped |
| Use the meta-teacher to design the training sequence | Lunar-Lander student training · Lunar-Lander | training sequence length: 12 -> 10.38 ± 0.11 | Helped |
| Use the meta-teacher rather than UED teachers to design finite training sequences | UED teachers · Bit-Flipping | training sequence length | Helped |
| Use the meta-teacher rather than UED methods to design finite training sequences | UED methods · Lunar-Lander | training sequence length | Helped |
| Use the meta-teacher to provide training sequences to transferred students | New students in Bit-Flipping · Bit-Flipping | generalization performance | Helped |
| Use the meta-teacher to provide training sequences to transferred students | New students in Lunar-Lander · Lunar-Lander | generalization performance | Helped |
| Use the meta-teacher to generate training sequences | Students in Minigrid · Minigrid | IQM | Helped |
| Use the meta-teacher to generate training sequences | Students in Minigrid · Minigrid | optimality gap | Helped |
| Stop student training once optimal performance is attained in validation environments | Meta-teacher training · Lunar-Lander | generalization performance | Helped |
Surprising findings
- In Lunar-Lander, student generalization performance decreases as the student's training amount increases, attributed to overfitting to training parameter values.
Limitations
- Experiments are proof-of-concept on simple domains (Bit-Flipping, Lunar-Lander, 5x5 Minigrid mazes).
- Minigrid teacher samples from a pre-generated DSAGE maze set rather than generating environments directly.
- Authors note future work will move to human-related training.
Experimental setup
Datasets: Bit-Flipping Lunar-Lander Minigrid DSAGE-generated mazes
Benchmarks: Bit-Flipping Lunar-Lander Minigrid
Metrics: normalized student generalization performance training sequence length IQM optimality gap
Baselines: Domain Randomization PAIRED PLR⊥ ACCEL human-designed training sequence
Builds on: Unsupervised Environment Design PAIRED PLR ACCEL PEARL
Closest papers by meaning
- Improving Regret Approximation for Unsupervised Dynamic Environment Generation (NeurIPS 2025 · similarity 0.89)
- No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery (NeurIPS 2024 · similarity 0.89)
- Marginal Benefit Driven RL Teacher for Unsupervised Environment Design (AAAI 2025 · similarity 0.88)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.88)
- Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity (NeurIPS 2024 · similarity 0.88)
- Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.