Research map / Reinforcement learning / Visual and representation RL
Improving Environment Novelty Quantification for Effective Unsupervised Environment Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CENIE, which scores environment novelty by the likelihood of a student's state-action trajectories under a GMM fit to its curriculum coverage buffer, and combines it with regret in PLR⊥ and ACCEL for unsupervised environment design.
Problem
Regret-based unsupervised environment design (UED) curricula increase difficulty but neglect environment novelty, and existing novelty measures are domain-specific, curriculum-agnostic, or computationally costly.
Approach
Quantify environment novelty via the student's state-action space coverage from prior curriculum experiences, modeled with a Gaussian Mixture Model (K chosen by silhouette score over a FIFO buffer), and use it jointly with regret to prioritize level replay in PLR⊥ and ACCEL.
Key results
- On BipedalWalker, ACCEL-CENIE reaches close to 55% of optimal performance, outperforming all benchmarks in aggregate IQM and optimality gap.
- On CarRacing's 20 F1 tracks, PLR-CENIE achieves a mean return of 553 versus 499 for DIPLR and 392 for PLR⊥.
- PLR-CENIE reaches near 50% solved rate on PerfectMazeLarge, versus under 10% for PLR⊥.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CENIE novelty prioritization to PLR⊥ | PLR⊥ · CarRacing F1 tracks (mean) | return: 392±28 -> 553±32 | Helped |
| Add CENIE novelty prioritization to PLR⊥ | PLR⊥ · BipedalWalker | state-action space coverage: 43.4% -> 55.3% | Helped |
| Add CENIE novelty prioritization to ACCEL | ACCEL · BipedalWalker | state-action space coverage: 42.5% -> 47.6% | Helped |
| Add CENIE novelty prioritization to ACCEL | ACCEL · Minigrid held-out environments | solved rate | Helped |
| Add CENIE novelty prioritization to PLR⊥ | PLR⊥ · Minigrid held-out environments | solved rate | Helped |
| Add CENIE novelty prioritization to ACCEL | ACCEL · BipedalWalker test environments | return | Helped |
| Add CENIE novelty prioritization to PLR⊥ | PLR⊥ · BipedalWalker test environments | return | Helped |
| Add CENIE novelty prioritization to PLR⊥ | PLR⊥ · CarRacing F1 tracks | return | Helped |
| Use novelty alone for replay prioritization (ablation) | PLR-CENIE · Minigrid held-out environments | solved rate | Helped |
| Use novelty alone for replay prioritization (ablation) | ACCEL-CENIE · Minigrid held-out environments | solved rate | Mixed |
| Use novelty alone for replay prioritization (ablation) | PLR-CENIE · BipedalWalker test environments | return | No clear effect |
| Use novelty alone for replay prioritization (ablation) | ACCEL-CENIE · BipedalWalker test environments | return | Hurt |
Surprising findings
- Novelty-only prioritization (PLR-CENIE†) matches or outperforms regret-based PLR⊥ in several settings, notably CarRacing.
- PLR-CENIE maintains comparable or slightly higher total regret in the replay buffer despite not directly optimizing regret.
Limitations
- Curse of dimensionality may hinder GMM-based density estimation in high-dimensional state-action spaces.
- Fixed 0.5-0.5 weighting between regret and novelty was not tuned.
- CENIE was not used to directly generate novel environments.
Experimental setup
Datasets: Minigrid BipedalWalker CarRacing
Benchmarks: Minigrid held-out mazes PerfectMazeLarge BipedalWalker test environments CarRacing F1 tracks
Metrics: solved rate return IQM optimality gap state-action space coverage
Baselines: Domain Randomization Minimax PAIRED DIPLR PLR⊥ ACCEL
Builds on: PLR⊥ ACCEL PAIRED POET DIPLR
Closest papers by meaning
- No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery (NeurIPS 2024 · similarity 0.91)
- Improving Regret Approximation for Unsupervised Dynamic Environment Generation (NeurIPS 2025 · similarity 0.90)
- Marginal Benefit Driven RL Teacher for Unsupervised Environment Design (AAAI 2025 · similarity 0.88)
- Reward-Free Curricula for Training Robust World Models (ICLR 2024 · similarity 0.86)
- Unsupervised Training Sequence Design: Efficient and Generalizable Agent Training (AAAI 2024 · similarity 0.85)
- Imagined Autocurricula (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.