Research map / Reinforcement learning / Visual and representation RL
Implicit Curriculum in Procgen Made Explicit
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds C-Procgen, a Procgen variant exposing explicit context parameters, and uses PPO runs to show that uniformly sampled multi-context training implicitly follows an easy-to-hard curriculum.
Problem
Procgen levels vary in difficulty through hidden, black-box context parameters, and prior curriculum work manipulates level order or generation; how learning itself progresses across diverse contexts under uniform multi-level training is under-studied.
Approach
Refactor Procgen to expose over 100 environment parameters so contexts are explicitly controllable, then analyze per-context score, loss, and sample counts over training to reveal an implicit curriculum emerging without level prioritization, plus a Loss Production Efficiency measure linked to PLR gains.
Key results
- Across nine Procgen games, loss and return shift from easy to hard contexts over training despite uniform context sampling.
- Lower Loss Production Efficiency correlates with larger score gains from Prioritized Level Replay across Procgen games (qualitative correlation shown in Figure 4).
- Expanding the context set (Leaper Setting 1 with six new contexts) accelerates the shift of loss focus toward harder contexts relative to the original setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with uniformly sampled contexts and no level prioritization | PPO on Procgen · Procgen games | learning progression across context difficulty | Helped |
| Use C-Procgen to expose and control environment parameters | Procgen · Procgen benchmark | context control | Helped |
| Mask out selected training contexts (ablation) | PPO on Procgen · Coinrun, Heist, Ninja, Climber, and other Procgen games | final average score | Mixed |
| Remove difficult contexts from training (ablation) | PPO on Procgen · Climber, Coinrun, and Heist | final average score | Helped |
| Expand the training context set with six contexts that connect the original Leaper contexts (ablation) | PPO on Leaper · Leaper | speed of loss shift toward hard contexts | Helped |
| Expand Leaper training to allow both lane parameters to take any value from {0, 1, 2, 3} (ablation) | PPO on Leaper · Leaper | learning speed | Helped |
| Use the original Leaper context setting rather than expanded contexts (ablation) | PPO on Leaper · Leaper | speed of loss shift toward hard contexts | Hurt |
| Use the original uniformly sampled context distribution | PPO on Procgen · Procgen games | loss production efficiency | Hurt |
Surprising findings
- Samples concentrate on hard contexts for most of training, while loss concentrates on moderately difficult contexts, a mismatch that may waste samples.
- Excluding some contexts sometimes improved final performance, e.g., in Climber setting 3 and Heist settings 2 and 3.
Limitations
- No specific evaluation protocol is provided for C-Procgen.
- Only Procgen games are covered, and dynamic context distributions evolving with learning were not explored.
- The LPE–PLR correlation is observed qualitatively across games and not established rigorously.
Experimental setup
Datasets: Procgen Benchmark
Benchmarks: Procgen Benchmark
Metrics: average score loss per sample sample count per context loss production efficiency score improvement with PLR
Baselines: Prioritized Level Replay
Builds on: Procgen PPO Prioritized Level Replay IMPALA
Closest papers by meaning
- Automated curriculum generation through setter-solver interactions (ICLR 2020 · similarity 0.88)
- Leveraging Procedural Generation to Benchmark Reinforcement Learning (ICML 2020 · similarity 0.87)
- Active Curriculum Refinement for Reinforcement Learning (ICML 2026 · similarity 0.87)
- PORTAL: Automatic Curricula Generation for Multiagent Reinforcement Learning (AAAI 2024 · similarity 0.85)
- Diffusion-based Curriculum Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- Learning Progress Driven Multi-Agent Curriculum (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.