Research map / Reinforcement learning / Offline RL
DIDI: Diffusion-Guided Diversity for Offline Behavioral Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIDI, which uses a pretrained diffusion model as a prior and a skill discriminator's diversity gradient to train a single-forward-pass contextual policy that learns diverse skills from label-free offline data.
Problem
Learning diverse, controllable behaviors from a mixture of label-free offline data, where existing offline methods collapse toward a single behavior and lack diversity encouragement.
Approach
Train a diffusion model on offline trajectories as a fixed prior, then regularize a latent-conditioned single-pass policy by back-propagating through the diffusion noising/denoising loss while jointly optimizing a skill discriminator's mutual-information (diversity) objective and an optional reward term; the diffusion noising step relabels pseudo-behaviors back onto the data manifold.
Key results
- On the diversity metric, DIDI scores 7.2 / 12.2 / 10.2 / 9.8 on Push T / Push F / Push 7 / Kitchen, above k-means-DI (6.8 / 10.4 / 8.1 / 8.7) and VAE-DI variants.
- On Push with obstacles after fine-tuning only skill embeddings, DIDI's success rises e.g. Push 7 IL (5 demos) from 0.30 to 0.88 versus a single fine-tuned policy.
- On D4RL with extrinsic reward, DIDI skills average 75.6 / 74.8 / 75.7 versus CQL 77.6, DT 74.7, and Diffuser 77.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DIDI instead of the strongest listed baseline | DIDI · D4RL HalfCheetah-medium-expert | normalized score: 91.6 vs 83.0, 81.9, 86.6 | Hurt |
| Use DIDI instead of the strongest listed baseline | DIDI · D4RL Hopper-medium-expert | normalized score: 107.6 vs 101.2, 102.1, 101.9 | Hurt |
| Use DIDI instead of the strongest listed baseline | DIDI · D4RL Walker2d-medium-expert | normalized score: 108.8 vs 107.5, 105.6, 106.9 | Hurt |
| Use DIDI instead of the strongest listed baseline | DIDI · D4RL HalfCheetah-medium | normalized score: 44.0 vs 43.0, 39.7, 42.3 | Mixed |
| Use DIDI instead of the strongest listed baseline | DIDI · D4RL Hopper-medium | normalized score: 74.3 vs 70.4, 74.1, 72.8 | Mixed |
| Use DIDI instead of the strongest listed baseline | DIDI · D4RL Walker2d-medium | normalized score: 79.6 vs 80.3, 78.3, 79.1 | Helped |
| Use DIDI instead of the strongest listed baseline | DIDI · D4RL HalfCheetah-medium-replay | normalized score: 45.5 vs 34.3, 33.2, 34.8 | Hurt |
| Use DIDI instead of the strongest listed baseline | DIDI · D4RL Hopper-medium-replay | normalized score: 95.0 vs 91.5, 87.8, 96.1 | Mixed |
| Use DIDI instead of the strongest listed baseline | DIDI · D4RL Walker2d-medium-replay | normalized score: 77.2 vs 68.9, 70.3, 68.1 | Hurt |
| Use diffusion-guided DIDI rather than k-means-DI, VAE-DI, or VAE-DI-fixed | DIDI · Push T | behavioral diversity: 6.8, 2.5, 1.6 -> 7.2 | Helped |
| Use diffusion-guided DIDI rather than k-means-DI, VAE-DI, or VAE-DI-fixed | DIDI · Push F | behavioral diversity: 10.4, 6.5, 7.1 -> 12.2 | Helped |
| Use diffusion-guided DIDI rather than k-means-DI, VAE-DI, or VAE-DI-fixed | DIDI · Push 7 | behavioral diversity: 8.1, 7.7, 5.4 -> 10.2 | Helped |
Surprising findings
- On D4RL, DIDI's reward-guided skills sometimes trail the standard offline baselines by only small margins while being more diverse, rather than clearly outperforming them.
Limitations
- The D4RL evaluation reports average returns only for three skills per task, and diversity on D4RL is shown qualitatively in appendix figures.
- Diversity evaluated with a variance-based metric whose definition is cited rather than fully specified in the text.
Experimental setup
Datasets: Push (IBC-derived) Kitchen (566 human demonstrations) Humanoid (PHC) D4RL halfcheetah/hopper/walker2d
Benchmarks: Push Kitchen Humanoid D4RL Gym-MuJoCo
Metrics: behavioral diversity (variance of motions across action type success rate normalized D4RL return
Baselines: k-means-DI VAE-DI VAE-DI-fixed CQL Decision Transformer Diffuser
Builds on: Diffuser DDPM IBC PHC
Closest papers by meaning
- Skill Disentanglement in Reproducing Kernel Hilbert Space (AAAI 2025 · similarity 0.87)
- Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment (ICLR 2025 · similarity 0.87)
- Robust Policy Learning via Offline Skill Diffusion (AAAI 2024 · similarity 0.87)
- Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts (ICML 2024 · similarity 0.86)
- Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Image Generation (NeurIPS 2026 · similarity 0.86)
- How to Leverage Diverse Demonstrations in Offline Imitation Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.