Research map / Reinforcement learning / Offline RL
Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CFDG, a classifier-free guidance diffusion data augmentation method that generates both offline and online synthetic data for offline-to-online RL, with reweighting toward online synthetic data, and integrates it into IQL, PEX and APL on D4RL.
Problem
Offline-to-online RL fine-tunes an offline-pretrained policy with limited online interaction, but data generated by existing augmentation methods (e.g., EDIS) from offline data still mismatches the online data distribution, limiting performance.
Approach
Treat offline and online data as two condition labels for a single classifier-free guidance diffusion model (EDM-based), generate both data types with one training process and no extra classifier, and reweight the batch to favor online synthetic data that better matches the current policy.
Key results
- CFDG improves IQL Locomotion total from 810 to 933 and PEX from 890 to 1024, and APL from 972 to 1081 across 12 Locomotion tasks.
- On AntMaze (IQL and PEX), CFDG raises totals from 250 to 266 (IQL) and 264 to 284 (PEX).
- On Adroit, CFDG averages 43.3 versus 26.6 for EDIS and 25.7 for the base IQL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CFDG to IQL | IQL · Locomotion total | normalized score: 810 -> 933 | Helped |
| Add CFDG to PEX | PEX · Locomotion total | normalized score: 890 -> 1024 | Helped |
| Add CFDG to APL | APL · Locomotion total | normalized score: 972 -> 1081 | Helped |
| Add CFDG to IQL | IQL · AntMaze total | normalized score: 250 -> 266 | Helped |
| Add CFDG to PEX | PEX · AntMaze total | normalized score: 264 -> 284 | Helped |
| Add CFDG to IQL | IQL · Adroit average | score: 25.7 -> 43.3 | Helped |
| Add classifier-free guidance (ablation) | CFDG · Locomotion average | normalized score: 74.13 -> 82.10 | Helped |
| Add classifier-free guidance (ablation) | CFDG · Locomotion, halfcheetah average | normalized score: 80.65 -> 84.44 | Helped |
| Add classifier-free guidance (ablation) | CFDG · Locomotion, hopper average | normalized score: 65.75 -> 68.22 | Helped |
| Add classifier-free guidance (ablation) | CFDG · Locomotion, walker2d average | normalized score: 75.99 -> 93.65 | Helped |
| Add offline data augmentation (ablation) | CFDG · Locomotion average | normalized score: 76.77 -> 82.10 | Helped |
| Add offline data augmentation (ablation) | CFDG · Locomotion, halfcheetah average | normalized score: 81.22 -> 84.44 | Helped |
Surprising findings
- Generated online data alone (SynthER-style augmentation of online data) outperformed EDIS's offline-data-based augmentation, supporting that online data is more policy-aligned.
Limitations
- Adds computational and time overhead from diffusion model training and sampling.
- Synthetic data ratio and reweighting (8:2 online/offline synthetic split, 1/3 ratio) are fixed hyperparameters chosen by the authors.
- Evaluated only on D4RL Locomotion, AntMaze (medium and large only) and Adroit tasks.
Experimental setup
Datasets: D4RL halfcheetah D4RL hopper D4RL walker2d D4RL antmaze Adroit relocate-human-v1 Adroit pen-human-v1 Adroit door-human-v1
Benchmarks: D4RL Locomotion D4RL AntMaze Adroit
Metrics: normalized return Jensen-Shannon divergence
Baselines: IQL PEX APL SynthER EDIS
Builds on: classifier-free guidance Elucidated Diffusion Model EDIS IQL PEX APL
Closest papers by meaning
- Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.91)
- Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (ICLR 2025 · similarity 0.90)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- The Generalization Gap in Offline Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL (NeurIPS 2024 · similarity 0.89)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.