Research map / Reinforcement learning / Offline RL
ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ADG, a three-stage diffusion framework that trains an ambient DDPM on partially corrupted offline RL data, uses its noise-prediction norm to detect corrupted samples, and refines them with a standard DDPM trained on the clean subset, then runs any offline RL algorithm on the recovered dataset.
Problem
Offline RL on real-world datasets with corrupted states, actions, and rewards degrades performance, especially for high-dimensional states and when multiple elements are corrupted; standard diffusion models overfit to noisy training samples.
Approach
Extends ambient diffusion (learning from approximated, noise-consistent distributions) to discrete DDPM, uses the noise-predictor's squared norm at timestep k_a to detect corruption, then trains a standard DDPM only on detected-clean trajectory slices to recover corrupted samples before offline RL.
Key results
- Under random state corruption, ADG raises average score across CQL, IQL, RIQL, DT, RDT, e.g., CQL from 10.6 to 15.4 and DT from 45.0 to 58.3 (reported improvements 29.56% for DT).
- Under random full-element corruption on 1% Adroit and 10% MuJoCo tasks, ADG raises CQL average from 0.9 to 10.2 and IQL from 19.0 to 34.0.
- Under adversarial state corruption, ADG improves average performance by about 24.41% across baselines, with IQL and DT with ADG outperforming robust variants RIQL and RDT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ADG dataset recovery before CQL | CQL · Random state corruption, all benchmarks | average score: 10.6 -> 15.4 | Helped |
| Apply ADG dataset recovery before IQL | IQL · Random state corruption, all benchmarks | average score: 24.9 -> 37.4 | Helped |
| Apply ADG dataset recovery before RIQL | RIQL · Random state corruption, all benchmarks | average score: 24.7 -> 40.2 | Helped |
| Apply ADG dataset recovery before DT | DT · Random state corruption, all benchmarks | average score: 45.0 -> 58.3 | Helped |
| Apply ADG dataset recovery before RDT | RDT · Random state corruption, all benchmarks | average score: 51.9 -> 57.8 | Helped |
| Use ambient diffusion loss instead of naive diffusion loss for detector training (ablation) | ADG detector · Kitchen-complete-v0 and relocate-expert-v1 with Random State Attack | false negative rate | Helped |
| Set detector timestep to tn = 30 rather than 15 or 50 (ablation) | ADG detector · Kitchen-complete-v0 and relocate-expert-v1 with Random State Attack | false negative rate | Helped |
| Use selective training with moderate ζ values (0.10 or 0.20) rather than ζ=0.05 or ζ=0.50 (ablation) | ADG denoiser · Walker2d-medium-replay-v2 with Random State Attack | MSE | Helped |
| Use two independent diffusion models rather than one shared model for detection and denoising (ablation) | ADG · Walker2d-medium-replay-v2 with Random State Attack | D4RL score | Helped |
| Increase slice-window length H from 0 (ablation) | ADG · Offline RL corruption benchmarks | performance | Helped |
Surprising findings
- Standard offline RL algorithms with ADG-recovered data can outperform their robust variants (IQL over RIQL, DT over RDT) in nearly all scenarios.
- Filtering corrupted samples alone is insufficient for sequence models, since removing noisy samples leaves incomplete trajectories; recovery is needed.
Limitations
- Detection threshold zeta and approximation timestep k_a require manual selection with a trade-off between information retention and approximation error.
- Evaluated only on D4RL benchmarks with synthetic corruption (random and adversarial), not real-world sensor or human-input noise.
Experimental setup
Datasets: D4RL MuJoCo medium-replay-v2 D4RL Kitchen D4RL Adroit expert-v0
Benchmarks: D4RL MuJoCo D4RL Kitchen D4RL Adroit
Metrics: normalized D4RL score average score improvement percentage MSE vs ground truth false negative rate
Baselines: CQL IQL RIQL DT RDT
Builds on: DDPM Ambient Diffusion CQL IQL Decision Transformer RIQL RDT
Closest papers by meaning
- Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling (ICLR 2025 · similarity 0.90)
- Towards Robust Offline Reinforcement Learning under Diverse Data Corruption (ICLR 2024 · similarity 0.90)
- Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption (NeurIPS 2025 · similarity 0.89)
- Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation (ICML 2025 · similarity 0.88)
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.