Research map / Reinforcement learning / Policy gradient and actor-critic
Diffusion Policy Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPPO, a framework that fine-tunes pre-trained diffusion policies with PPO by treating the denoising chain as an inner MDP nested in the environment MDP, with practical design choices for stable fine-tuning.
Problem
Fine-tuning diffusion-based robot policies with reinforcement learning is considered hard because marginal action likelihoods are intractable and policy gradient was conjectured to be inefficient through the multi-step denoising process.
Approach
Model the denoising process as a Diffusion Policy MDP nested within the environment MDP, where each denoising step has an explicit Gaussian likelihood, then apply PPO with a denoising-discounted advantage that depends only on environment state, plus practices such as fine-tuning only the last denoising steps or DDIM, and clipping the noise schedule for exploration.
Key results
- On Robomimic Transport, DPPO is reported as the first RL method to reach >50% success from state or pixel input, exceeding 90% with state input.
- On zero-shot sim-to-real One-leg assembly, DPPO reaches 80% (16/20 trials) on hardware while the Gaussian policy gets 0%.
- DPPO is reported to be consistently strong on Gym and Robomimic compared with diffusion-based RL baselines, with off-policy baselines failing in sparse-reward Robomimic tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune a pre-trained Diffusion Policy with DPPO rather than Gaussian-MLP | Robomimic Transport with state input · Robomimic Transport state | success rate | Helped |
| fine-tune a pre-trained Diffusion Policy with DPPO rather than Gaussian-ViT-MLP | Robomimic Transport with pixel input · Robomimic Transport pixel | success rate | Helped |
| use DPPO instead of Gaussian-MLP for zero-shot hardware deployment | Furniture-Bench One-leg · Furniture-Bench One-leg sim-to-real | hardware success rate: 0% -> 80% | Helped |
| add an auxiliary behavior-cloning loss during Gaussian fine-tuning | Gaussian policy on Furniture-Bench One-leg · Furniture-Bench One-leg sim-to-real | hardware success rate: 0% -> 50% | Mixed |
| use an environment-state-only value estimator instead of estimating value from the state and denoised action (ablation) | DPPO · Robomimic Can and Square | fine-tuning performance | Helped |
| estimate value from the state and denoised action rather than environment state alone (ablation) | DPPO · Hopper-v2 | reward | Helped |
| use denoising discount factor 0.8 rather than other tested values (ablation) | DPPO · Hopper-v2 and HalfCheetah-v2 | training efficiency | Helped |
| use denoising discount factor 0.5 (ablation) | DPPO · Hopper-v2 and HalfCheetah-v2 | training speed | Hurt |
| vary the denoising discount factor (ablation) | DPPO · Robomimic Can | training performance | No clear effect |
| use denoising discount factor 0.5 (ablation) | DPPO · Robomimic Square | training performance | Helped |
| fine-tune only the last few denoising steps rather than all steps (ablation) | DPPO · robotics and control tasks | runtime | Helped |
| use a clipped higher minimum diffusion noise level for exploration (ablation) | DPPO · DPPO fine-tuning tasks | training performance | Mixed |
Surprising findings
- Policy gradient through a diffusion policy's full denoising chain, conjectured to be inefficient, performs strongly when fine-tuning from pre-trained policies.
- Off-policy demo-augmented methods (RLPD, Cal-QL) fail to learn at all on Robomimic Can and Square, while DPPO reaches near-100% success.
Limitations
- Lower sample efficiency than off-policy methods, e.g. on HalfCheetah-v2 demo-augmented baselines are faster.
- Fine-tuning with MLP heads with T_a < T_p was unstable; UNet needed for variable chunk length.
- Structured on-manifold exploration may hinder fine-tuning when aggressive unstructured exploration is needed (e.g., Lamp with Low randomness).
Experimental setup
Datasets: D4RL Robomimic FurnitureBench D3IL
Benchmarks: OpenAI Gym Hopper-v2 Walker2D-v2 HalfCheetah-v2 Robomimic Lift Can Square Transport FurnitureBench One-leg Lamp Round-table D3IL Avoid
Metrics: success rate average reward
Baselines: Gaussian-MLP GMM DIPO IDQL DQL QSM RLPD Cal-QL IBRL DAWR DRWR
Builds on: Diffusion Policy PPO DDIM Generalized Advantage Estimation
Closest papers by meaning
- GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation (ICML 2025 · similarity 0.90)
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.90)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.90)
- Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.90)
- Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transport (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.