Research map / Reinforcement learning / Offline RL
Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EDIS, which trains a diffusion model on offline data and uses contrastive energy functions to guide sampling toward the online distribution, generating synthetic samples to augment online fine-tuning of Cal-QL and IQL.
Problem
Offline-to-online RL reuses offline data directly in the online phase, causing distribution shift and inefficient fine-tuning; model-based rollouts suffer compounding error.
Approach
Use a diffusion model fit on offline data and decompose the online distribution into state, action and transition energy terms learned by contrastive (InfoNCE) energy prediction, so energy-guided diffusion sampling generates on-policy-like synthetic transitions for online training without modeling the transition function as a rollout model.
Key results
- EDIS on Cal-QL raises total D4RL score from 1448.5 to 1830.1 after 0.2M online fine-tuning (about 26% improvement).
- EDIS on IQL raises total score from 1244.2 to 1395.1, an average gain of about 12%.
- Averaged over Cal-QL and IQL, the paper reports roughly 20% average improvement across MuJoCo, AntMaze and Adroit.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add EDIS to Cal-QL | Cal-QL · All benchmarks total | score: 1448.5 -> 1830.1 | Helped |
| Add EDIS to IQL | IQL · All benchmarks total | score: 1244.2 -> 1395.1 | Helped |
| Add state-energy guidance (ablation) | EDIS · hopper-random-v2 | state divergence: 0.85$\pm$0.02 -> 0.73$\pm$0.03 | Helped |
| Add action-energy guidance (ablation) | EDIS · hopper-random-v2 | action divergence: 0.51$\pm$0.04 -> 0.39$\pm$0.02 | Helped |
| Add transition-energy guidance (ablation) | EDIS · hopper-random-v2 | transition divergence: 0.69$\pm$0.03 -> 0.66$\pm$0.04 | Helped |
Surprising findings
- An MLP transition model used for rollout does not improve over the base Cal-QL, while a diffusion transition model gives some gain but still falls short of EDIS.
- Without the state energy function, EDIS struggles on hopper-random, indicating that merging unrelated offline data directly hurts fine-tuning.
Limitations
- Theoretical bounds rely on Fitted-Q-iteration assumptions (realizability, closure under Bellman update, Lipschitz function class) and a single-iteration analysis.
- Experiments are limited to D4RL benchmarks with low-dimensional state and action spaces; scalability to high-dimensional settings is not tested.
Experimental setup
Datasets: D4RL hopper, halfcheetah, walker2d (v2) D4RL antmaze (v2) D4RL adroit (human-v1)
Benchmarks: D4RL MuJoCo locomotion D4RL AntMaze navigation D4RL Adroit manipulation
Metrics: normalized return JS divergence mean squared error of action and next state
Baselines: Cal-QL IQL MLP model-based rollout diffusion transition model rollout
Builds on: Cal-QL IQL classifier guidance diffusion (energy-guided score sampling) InfoNCE Diffuser
Closest papers by meaning
- Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation (ICML 2025 · similarity 0.91)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (ICLR 2025 · similarity 0.89)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Information-Directed Offline-to-Online Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Planning, Fast and Slow: Online Reinforcement Learning with Action-Free Offline Data via Multiscale Planners (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.