Research map / Reinforcement learning / Imitation learning
Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transport
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OTPR, which fine-tunes a score-based diffusion policy with RL by treating the policy as an L2-regularized optimal transport map with Q-function cost, using expert-keypoint masked OT and compatibility-based resampling.
Problem
Diffusion policies learned from demonstrations are brittle under distribution shift, and standard RL fine-tuning of iterative diffusion models is unstable, computationally costly, or architecture-specific.
Approach
Use the Q-function as an optimal transport cost and derive a compatibility function H from the L2-regularized OT dual; train the diffusion score on H-weighted (soft-coupled) state-action pairs, add expert keypoint masks to the OT plan, and resample actions from the policy by compatibility weights during online fine-tuning.
Key results
- On Franka-Kitchen Kitchen-Complete-v0, OTPR improves from 61 to 92, versus Cal-QL 19 to 57 and IBRL 0 to 25.
- On RoboMimic Can-State, OTPR reaches 99 after fine-tuning (from 63), and Square-State reaches 98 (from 40), compared with IBRL's 64 and 50.
- Learning curves reportedly show OTPR dominates or matches the next best method, while IDQL, DQL, and DPPO are unstable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use compatibility-function guidance H instead of Q-value guidance (ablation) | OTPR · Robomimic-Can | evaluation performance | Helped |
| Use compatibility-function guidance H instead of advantage guidance A (ablation) | OTPR · Robomimic-Can | evaluation performance | Helped |
| Add the expert-data mask to optimal transport (ablation) | OTPR · Robomimic-Can | training stability and efficiency | Helped |
Surprising findings
- Without the expert mask (OTPR-U), the method still functions as an offline RL algorithm using the compatibility function alone.
Limitations
- Relies on a Q-network critic and an approximate dual potential estimation, with bounds that depend on epsilon-approximate maximizers and the W2 bound is only sketched.
- The Wasserstein bound for the SDE sampler error is not derived explicitly and is only interpreted from existing convergence results.
Experimental setup
Datasets: Robomimic Franka-Kitchen CALVIN
Benchmarks: Robomimic Franka-Kitchen CALVIN
Metrics: success rate normalized score max sub-tasks completed
Baselines: IDQL DPPO DQL RLPD Cal-QL IBRL
Builds on: Implicit Q-Learning Diffusion Policy DPPO Cal-QL L2-regularized optimal transport
Closest papers by meaning
- Efficient Online Reinforcement Learning for Diffusion Policy (ICML 2025 · similarity 0.90)
- Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.90)
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.90)
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.89)
- Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.89)
- Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.