Research map / Reinforcement learning / Offline RL
ReDiffuser: Reliable Decision-Making Using a Diffuser with Confidence Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReDiffuser, which adds an RND-based confidence function to Diffuser for offline RL, using it to guide sampling toward reliable decisions via adaptive-horizon and value-embedded planning.
Problem
Diffusion-based offline RL planners produce non-deterministic decisions whose performance is unstable and whose reliability cannot be measured, since conformal-style guarantees need ground truth unavailable in RL.
Approach
Train a Random Network Distillation predictor on historical decisions sampled from a pretrained Diffuser to get a confidence score, then add the confidence as a guidance term to the diffusion sampling distribution; collect training trajectories with diverse horizons (goal-conditioned) or only high-value trajectories (reward-maximization) to realize adaptive-horizon and value-embedded planning.
Key results
- Maze2D Multi-Large score rises from 142.3 (Diffuser) to 176.1 with ReDiffuser; average Maze2D score 160.1 vs 133.4 for Diffuser.
- Locomotion average normalized score 82.6 vs 80.3 for Diffuser, ranked first among compared methods.
- KUKA block stacking success improves from 60.5 (Diffuser) to 67.0 with naive RND confidence estimation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RND-based confidence estimation | Diffuser · KUKA block stacking | normalized score: 60.5 ±1.1 -> 67.0 ±1.5 | Helped |
| Add adaptive-horizon planning | Diffuser · Maze2D Single-Umaze | score: 121.8±0.6 -> 145.3 ±1.2 | Helped |
| Add adaptive-horizon planning | Diffuser · Maze2D Single-Medium | score: 130.7±0.8 -> 140.3 ±1.8 | Helped |
| Add adaptive-horizon planning | Diffuser · Maze2D Single-Large | score: 130.0±1.7 -> 159.6 ±3.0 | Helped |
| Add adaptive-horizon planning | Diffuser · Maze2D Multi-Large | score: 142.3 ±1.6 -> 176.1 ±3.4 | Helped |
| Add value-embedded planning | Diffuser · Hopper Medium | score: 92.9 ±12.9 -> 100.2 ±6.0 | Helped |
| Add value-embedded planning | Diffuser · HalfCheetah Medium | score: 45.2 ±0.5 -> 45.2 ±0.7 | No clear effect |
| Set confidence proportion β to 20.0 rather than 0.0 (ablation) | ReDiffuser · KUKA block stacking | normalized score: 60.5 ±1.1 -> 67.0 ±1.5 | Helped |
| Set confidence discount power p to 0.4 rather than 0.0 (ablation) | ReDiffuser · Maze2D Single-Medium | score: 81.5 ±4.9 -> 140.3 ±1.8 | Helped |
| Set confidence discount power p to 0.6 rather than 0.0 (ablation) | ReDiffuser · Maze2D Single-Large | score: 89.2 ±6.3 -> 159.6 ±3.0 | Helped |
Limitations
- Offline method without exploration capability; performance bounded by offline dataset quality.
- Validated only on simple navigation and locomotion tasks plus one manipulation task.
- No real-world or sim-to-real validation yet.
Experimental setup
Datasets: D4RL Maze2D D4RL MuJoCo locomotion (Medium-Expert, Medium, Medium-Replay) KUKA block stacking (PDDLStream expert data)
Benchmarks: D4RL Maze2D D4RL locomotion KUKA block stacking
Metrics: normalized score (mean and bias over seeds)
Baselines: Diffuser BCQ CQL IQL MPPI DT TT MOPO MOReL MBOP BC
Builds on: Diffuser Random Network Distillation BCQ Decision Diffuser
Closest papers by meaning
- Neural Network Approximation for Pessimistic Offline Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning (ICML 2026 · similarity 0.87)
- Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.86)
- Blending Imitation and Reinforcement Learning for Robust Policy Improvement (ICLR 2024 · similarity 0.86)
- A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.