Research map / Reinforcement learning / Visual and representation RL
Diffusion-based Curriculum Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiCuRL, which trains a conditional diffusion model online to generate curriculum goals guided by a Q-function and an AIM reward, and shows it matches or beats nine CRL baselines on maze and MuJoCo robot tasks.
Problem
Curriculum RL needs intermediate goals that are challenging yet achievable, but existing CRL methods often struggle to produce them in complex environments without domain knowledge.
Approach
A conditional diffusion model generates curriculum goals online from replay-buffer states; its training loss adds a Q-function term and an AIM (Wasserstein-based) reward term so generated goals are feasible and move toward the desired goal, while the noising/denoising process promotes exploration. A bipartite min-cost max-flow step selects the final goal.
Key results
- On PointUMaze, DiCuRL reaches success rate 1.0 in 28333 ± 3036 timesteps versus 29800 ± 4166 for OUTPACE and 263431 ± 114795 for GRADIENT.
- On PointSpiralMaze, DiCuRL reaches success rate 1.0 in 305833 ± 43225 timesteps versus 396875 ± 111451 for OUTPACE; most other baselines never reach it within 1e6 timesteps.
- Ablation on PointSpiralMaze shows the agent fails to achieve the task without the Q-function and performs worse without the AIM reward.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DiCuRL instead of OUTPACE | SAC curriculum learning · PointUMaze | timesteps to success rate 1.0: 29800 ± 4166 -> 28333 ± 3036 | Helped |
| use DiCuRL instead of OUTPACE | SAC curriculum learning · PointNMaze | timesteps to success rate 1.0: 113333 ± 24267 -> 108428 ± 34718 | Helped |
| use DiCuRL instead of OUTPACE | SAC curriculum learning · PointSpiralMaze | timesteps to success rate 1.0: 396875 ± 111451 -> 305833 ± 43225 | Helped |
| add the AIM reward rφ to curriculum goal generation (ablation) | DiCuRL · PointSpiralMaze | success rate | Helped |
| add the Q-function Qϕ to curriculum goal generation (ablation) | DiCuRL · PointSpiralMaze | success rate | Helped |
| condition curriculum generation on the final episode state sT (ablation) | DiCuRL · PointSpiralMaze | success rate | Hurt |
| use DiCuRL instead of HGG | SAC curriculum learning · PointUMaze | timesteps to success rate 1.0: 48750 ± 24314 -> 28333 ± 3036 | Helped |
Surprising findings
- Using only the final state s_T as diffusion input (the sT strategy) converges later than using sampled replay-buffer minibatches, showing the final state is not always closer to the goal.
Limitations
- The AIM reward may underperform in higher-dimensional settings, which could hinder goal generation for image-like data.
- The bipartite min-cost max-flow selection strategy may be suboptimal; alternatives could be more effective.
- Evaluated only on low-dimensional maze and two Fetch manipulation tasks.
Experimental setup
Datasets: PointUMaze (MuJoCo) PointNMaze (MuJoCo) PointSpiralMaze (MuJoCo) FetchPush FetchPickAndPlace
Benchmarks: PointUMaze PointNMaze PointSpiralMaze FetchPush FetchPickAndPlace
Metrics: success rate timesteps to reach success rate 1.0 timesteps to reach success rate 0.3
Baselines: ACL GoalGAN HGG ALP-GMM VDS PLR CURROT GRADIENT OUTPACE HER
Builds on: Soft Actor-Critic Adversarial Intrinsic Motivation OUTPACE Hindsight Goal Generation denoising diffusion probabilistic models
Closest papers by meaning
- Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models (ICLR 2026 · similarity 0.88)
- Learning to Reach Goals via Diffusion (ICML 2024 · similarity 0.88)
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Planner (ICML 2025 · similarity 0.87)
- Text-Aware Diffusion for Policy Learning (NeurIPS 2024 · similarity 0.87)
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.