Research map / Reinforcement learning / Offline RL
Reasoning with Latent Diffusion in Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LDCQ, which learns a beta-VAE latent space of temporally abstract trajectory segments, models the behavior prior with a latent diffusion model, and performs batch-constrained Q-learning over latents to avoid extrapolation error on D4RL.
Problem
Offline RL must stitch suboptimal trajectory pieces without extrapolation error from out-of-support actions, and existing methods struggle with multi-modal data and long-horizon sparse-reward credit assignment.
Approach
Model in-support H-step trajectory segments as latent skills with a state-conditioned latent diffusion prior, then learn a Q-function over (state, latent) pairs with batch-constrained targets sampled from the diffusion prior, and execute the chosen latent via an autoregressive low-level decoder.
Key results
- LDCQ reaches 150.1 on maze2d-large-v1 and 57.7 on antmaze-large-diverse-v2, versus 123.0 and 22.0 for Diffuser; LDGC reaches 206.8 and 73.6.
- On kitchen-partial-v0, LDCQ scores 67.8 versus 57.0 for Decision Diffuser and 50.1 for CQL.
- On locomotion, LDCQ is competitive but not best; e.g., walker2d-medium-v2 69.4 versus 82.5 for Decision Diffuser.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use goal-conditioned latent diffusion instead of LDCQ | LDCQ · maze2d-large-v1 | D4RL score: 150.1 $\pm$ 2.9 -> 206.8 $\pm$ 3.1 | Helped |
| Use goal-conditioned latent diffusion instead of LDCQ | LDCQ · antmaze-medium-diverse-v2 | D4RL score: 68.9 $\pm$ 0.7 -> 75.6 $\pm$ 0.9 | Helped |
| Use goal-conditioned latent diffusion instead of LDCQ | LDCQ · antmaze-large-diverse-v2 | D4RL score: 57.7 $\pm$ 1.8 -> 73.6 $\pm$ 1.3 | Helped |
| Use temporal abstraction with increasing sequence horizon H (ablation) | LDCQ · kitchen-mixed-v0 | D4RL score | Mixed |
| Use latent diffusion rather than a conditional Gaussian VAE prior to model multimodal latents | LDCQ · kitchen-mixed-v0 | D4RL score | Helped |
| Add 10 additional diffusion denoising steps at t=0 (ablation) | LDCQ · D4RL tasks | D4RL score | Helped |
| Increase diffusion steps T during evaluation (ablation) | LDCQ · D4RL locomotion tasks | D4RL score | Helped |
| Increase diffusion steps T beyond the initial upward trend (ablation) | LDCQ · Long-horizon tasks | D4RL score | No clear effect |
| Use temporal abstraction with increasing horizon H (ablation) | LDCQ · Locomotion tasks | D4RL score | No clear effect |
| Use LDCP planning instead of LDCQ | LDCQ · antmaze-medium-diverse-v2 | D4RL score: 68.9 $\pm$ 0.7 -> 77.0 $\pm$ 1.1 | Helped |
| Use LDCP planning instead of LDCQ | LDCQ · antmaze-large-diverse-v2 | D4RL score: 57.7 $\pm$ 1.8 -> 59.7 $\pm$ 1.3 | Helped |
Surprising findings
- A VAE Gaussian prior sampled latents out of support between modes, yielding a -66 average reward on the 1D random walk while the diffusion prior reached -2.2.
- Increasing temporal abstraction horizon H helps on kitchen but not on locomotion tasks.
Limitations
- Inference is slow due to iterative diffusion sampling with simple DDPM.
- Average performance on locomotion tasks, with no perturbation function to improve medium datasets.
- Sequence horizon H is fixed across an experiment rather than adaptive.
Experimental setup
Datasets: D4RL maze2d-large-v1 D4RL antmaze-medium-diverse-v2 D4RL antmaze-large-diverse-v2 D4RL kitchen-partial-v0 D4RL kitchen-mixed-v0 D4RL halfcheetah/walker2d/hopper locomotion D4RL Adroit D4RL CARLA
Benchmarks: D4RL maze2d-large-v1 D4RL antmaze-medium-diverse-v2 D4RL antmaze-large-diverse-v2 D4RL kitchen-partial-v0 D4RL kitchen-mixed-v0 D4RL locomotion D4RL Adroit D4RL CARLA
Metrics: D4RL normalized score
Baselines: BC BCQ CQL IQL Decision Transformer Diffuser Decision Diffuser BCQ-H
Builds on: Batch-Constrained Q-learning Latent diffusion models Diffuser Decision Diffuser Min-SNR-gamma
Closest papers by meaning
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models (NeurIPS 2025 · similarity 0.91)
- Pessimistic Latent Task-aware Optimization for Robust Offline Meta-Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Efficient Planning with Latent Diffusion (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.