Research map / Reinforcement learning / Visual and representation RL

Adversarial Environment Design via Regret-Guided Diffusion Models

NeurIPS 2024 · Reinforcement learning › Visual and representation RL · approach: Scalable RL systems and environments · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ADD, a unsupervised environment design method that guides a pre-trained diffusion-based environment generator with a differentiable regret estimate from a distributional environment critic, yielding zero-shot gains on maze and bipedal walker tasks.

Problem

Unsupervised environment design (UED) for training RL agents robust to environment changes; existing learning-based generators are unstable and have limited coverage, while replay-based methods are sample-inefficient and cannot directly generate meaningful environments.

Approach

Augment the UED regret objective with an entropy term (soft UED), whose optimal environment distribution is exp-weighted by regret; sample from it by regret-guided diffusion, where regret is made differentiable via an environment critic predicting a categorical return distribution and a CVaR-minus-mean approximation of max return.

Key results

• On 12 maze test environments, ADD reaches 0.85 mean solved rate vs 0.72 for ACCEL (the strongest baseline) and 0.64 for DR.
• ADD achieves the highest mean return of 149.6 across six BipedalWalker test environments, with 127.4 using half the environment steps.
• ADD w/o guidance reaches 0.63 mean solved rate, close to DR, suggesting regret guidance drives the gain.

Measured effects reported in the paper

Surprising findings

• PAIRED shows lower mean solved rate than DR on the maze tasks, indicating learning-based UED struggles in practice.
• ACCEL performs worse than PLR-perp on BipedalWalker despite being built on it, attributed to lower sample efficiency.

Limitations

• Convergence to the proposed optimal point is not guaranteed.
• The regret estimate is not tightly bounded relative to true regret.
• Critic is updated from episodic results that lag the current policy.

Experimental setup

Datasets: 10M randomly generated maze environments (13x13x3) randomly generated bipedal walker environment parameters human-designed test environments (Dennis et al., Jiang et al., Minigrid)

Benchmarks: Minigrid 12 zero-shot maze test environments BipedalWalker 6 zero-shot test environments

Metrics: solved rate average return interquartile range number of blocks shortest path length

Baselines: DR PAIRED PLR-perp ACCEL ADD w/o guidance

Builds on: PAIRED PLR ACCEL classifier guidance DDPM

Closest papers by meaning

• Improving Regret Approximation for Unsupervised Dynamic Environment Generation (NeurIPS 2025 · similarity 0.91)
• No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery (NeurIPS 2024 · similarity 0.90)
• Refining Minimax Regret for Unsupervised Environment Design (ICML 2024 · similarity 0.89)
• Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Marginal Benefit Driven RL Teacher for Unsupervised Environment Design (AAAI 2025 · similarity 0.88)
• DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design (ICML 2024 · similarity 0.88)