Research map / Reinforcement learning / RL from feedback
Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Planner
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SODP, a two-stage diffusion planner that pre-trains on task-agnostic sub-optimal multi-task trajectories without reward labels, then fine-tunes per task with PPO-style policy gradient plus a behavior-cloning regularizer.
Problem
Multi-task diffusion planners need task-specific expert demonstrations or reward labels, which are costly; the goal is a versatile planner learned from large-scale sub-optimal, task-agnostic trajectories that adapts quickly to downstream tasks.
Approach
Pre-train a guidance-free diffusion policy that predicts future action sequences from historical states on mixed sub-optimal multi-task data, then fine-tune it per task by treating the denoising chain as an MDP with PPO-style clipped importance-sampled policy gradients, regularized by a behavior-cloning loss on the best recent trajectories rather than KL or pre-training loss.
Key results
- SODP reaches 60.56% average success rate on Meta-World MT50-rand with sub-optimal data, above HarmoDT-F (57.20%) and MTDIFF-P (48.67%).
- With only 100k online fine-tuning steps per task, SODP achieves 59.26% vs. 57.37% for HarmoDT-F and 27.06% for MTDIFF-P.
- On image-based Adroit (3 tasks), SODP averages 73.9% vs. 68.3% for DP3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SODP instead of the strongest multi-task baseline | Meta-World MT50-rand · Meta-World MT50-rand | success rate: 57.20_{\pm 0.73} -> 60.56_{\pm 0.14} | Helped |
| reduce online fine-tuning to 100k steps | SODP · Meta-World MT50-rand | success rate: 60.56_{\pm 0.14} -> 59.26_{\pm 0.18} | Hurt |
| pre-train on multi-task MT-10 data rather than basketball only (ablation) | SODP · Unseen tasks: drawer-open, plate-slide-side, handle-pull-side | success rate: 34.7_{\pm 0.06} vs 0.0_{\pm 0.0}; 55.3_{\pm 0.33} vs 0.0_{\pm 0.0}; 71.3_{\pm 0.13} vs 0.0_{\pm 0.0} | Helped |
| remove pre-training and train from scratch (ablation) | SODP · Meta-World downstream tasks | success rate | Hurt |
| replace online reward fine-tuning with high-quality offline fine-tuning (ablation) | SODP · Meta-World MT-10, average across five tasks | success rate: 64.73_{\pm 0.19} -> 56.53_{\pm 0.18} | Hurt |
| remove pre-training and train only with high-quality offline data (ablation) | SODP · Meta-World MT-10, average across five tasks | success rate: 64.73_{\pm 0.19} -> 25.6_{\pm 0.18} | Hurt |
| replace reward-guided fine-tuning with direct BC (ablation) | SODP · Meta-World button-press-topdown, basketball, stick-pull | success rate: 60.7_{\pm 0.03} vs 51.3_{\pm 0.05}; 41.2_{\pm 0.16} vs 21.3_{\pm 0.03}; 50.5_{\pm 0.04} vs 26.7_{\pm 0.08} | Hurt |
| use stochastic rather than deterministic policy | MT-ODT · Meta-World MT-10 | success rate: 40.07_{\pm 0.15} -> 50.47_{\pm 0.21} | Helped |
| add BC regularization during fine-tuning (ablation) | SODP · Adroit image-based | success rate: 43.7 -> 73.9 | Helped |
| remove BC regularization during fine-tuning (ablation) | SODP · Meta-World downstream tasks | success rate | Hurt |
| replace BC regularization with KL regularization (ablation) | SODP · Meta-World downstream tasks | success rate | Hurt |
| replace BC regularization with diffusion pre-training loss (PL) (ablation) | SODP · Meta-World downstream tasks | success rate | Hurt |
Surprising findings
- KL and pre-training-loss regularizers confine fine-tuning to sub-optimal regions, performing worse than the proposed BC regularizer.
- A planner pre-trained only on a single task gets 0% on unseen tasks, while the multi-task pre-trained one reaches 34.7-71.3%.
Limitations
- Pre-training data quality matters: near-optimal data yields much higher success (80.93% vs 65.27% on MT10), so the sub-optimal setting limits performance.
- Fine-tuning requires online environment interaction per downstream task with task-specific rewards.
- Evaluated only on Meta-World and Adroit simulated manipulation tasks.
Experimental setup
Datasets: Meta-World sub-optimal SAC replay buffer data (50 tasks, 1M transitions/task) Adroit VRL3 replay buffer data (30% initial experiences)
Benchmarks: Meta-World MT50-rand Adroit Meta-World MT10 image-based
Metrics: average success rate learning curves
Baselines: Cal-QL IBRL RLPD MTSAC MTBC MTIQL MTDQL MTDT Prompt-DT MTDIFF HarmoDT DP3
Builds on: Diffusion-QL DPOK PPO DDPM DP3
Closest papers by meaning
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.89)
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Simple Hierarchical Planning with Diffusion (ICLR 2024 · similarity 0.89)
- Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transport (ICML 2025 · similarity 0.89)
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.