Research map / Reinforcement learning / Offline RL
Score Regularized Policy Optimization through Diffusion Behavior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SRPO, which extracts a deterministic offline-RL policy by regularizing the Q-gradient with the score of a pretrained diffusion behavior model, avoiding diffusion sampling at training and inference.
Problem
Diffusion behavior policies in offline RL are expressive but slow to sample (tens to hundreds of denoising steps per action), which hinders training, evaluation, and deployment in latency-sensitive settings.
Approach
Derive that the gradient of the reverse-KL behavior regularizer equals the score of the behavior distribution, estimate that score with a pretrained diffusion model, and regularize the policy gradient directly instead of the loss. Combined with implicit Q-learning, the policy is a deterministic MLP, and a DreamFusion-style multi-timestep ensemble and baseline term are used for variance reduction.
Key results
- SRPO averages 87.1 on D4RL locomotion, versus 88.0 for Diffusion-QL and 86.6 for QGPO, and 73.6 on AntMaze versus 79.1 for IDQL.
- SRPO boosts action sampling speed by more than 25x over leading diffusion-based methods on locomotion tasks, per the abstract.
- SRPO outperforms Gaussian/Dirac-policy baselines such as IQL and TD3+BC on most locomotion tasks, with averages of 87.1 vs 76.9 (IQL) and 75.3 (TD3+BC).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SRPO instead of diffusion-based baselines for action inference | D4RL locomotion tasks · D4RL locomotion | action sampling speed: 25 to 1000 times faster | Helped |
| Use SRPO instead of diffusion-based baselines for evaluation | D4RL locomotion tasks · D4RL locomotion | computational FLOPS: 0.25% to 0.01% | Helped |
| Replace behavior-candidate selection with an extracted deterministic policy | IDQL-style critic and behavior models · D4RL locomotion | normalized score | Mixed |
| Ensemble behavior scores from multiple diffusion times rather than using only the near-zero time score (ablation) | SRPO policy extraction · D4RL tasks | performance | Helped |
| Subtract the action-noise baseline from the estimated behavior gradient (ablation) | SRPO policy extraction · D4RL tasks | performance | Helped |
| Use ω(t)=σ_t² rather than a single-time weighting (ablation) | SRPO policy extraction · D4RL tasks | performance | Helped |
| Vary the weighting function ω(t) (ablation) | SRPO policy extraction · AntMaze and locomotion tasks | performance | Mixed |
| Use a deeper policy network for AntMaze (ablation) | SRPO policy extraction · AntMaze tasks | performance | Helped |
| Normalize β by the norm of the Q-gradient (ablation) | SRPO policy extraction · AntMaze tasks | training stability | Helped |
Surprising findings
- Using omega(t) concentrated at a single small diffusion time nearly recovers the original objective, but ensembling diffusion times was found empirically helpful for optimization, even though it biases the objective.
- Ensembling diffusion times and the baseline term are empirically beneficial for Antmaze tasks, which are sensitive to the weighting function, while locomotion tasks are not.
Limitations
- Surrogate objective does not converge to the exact optimal policy pi* because of the Q-term and finite policy expressivity, as stated in the remark.
- Temperature beta varies widely across tasks and requires per-task sweeps (Table of chosen beta values).
- AntMaze performance trails IDQL and QGPO, and Antmaze needed extra stabilization (normalized beta and deeper policy).
Experimental setup
Datasets: D4RL HalfCheetah D4RL Hopper D4RL Walker2d D4RL AntMaze
Benchmarks: D4RL locomotion (medium-expert, medium, medium-replay) D4RL AntMaze (umaze, medium, large)
Metrics: normalized return action sampling speed FLOPS
Baselines: BEAR TD3+BC IQL SfBC Diffuser Diffusion-QL QGPO IDQL
Builds on: implicit Q-learning continuous-time diffusion behavior modeling DreamFusion Diffusion-QL IDQL
Closest papers by meaning
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.93)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.92)
- Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.92)
- Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.90)
- Learning a Diffusion Model Policy from Rewards via Q-Score Matching (ICML 2024 · similarity 0.90)
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.