Research map / Reinforcement learning / RL from feedback
Score as Action: Fine Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes score-based diffusion models via continuous-time RL by treating score functions as actions, with a KL-regularized policy optimization (continuous-time PPO) and a structured value network, applied to CIFAR-10 and Stable Diffusion v1.5 with ImageReward.
Problem
Existing RL fine-tuning methods for diffusion models use discrete-time MDP formulations, which incur discretization errors, overfit to the sampling discretization, and are ill-suited to high-order or black-box solvers.
Approach
Treat the score function as a continuous control in a time-reversed SDE, formulate KL-regularized reward maximization as continuous-time stochastic control, and derive a continuous-time policy gradient and PPO-style update, with a value network that reuses denoiser (x-prediction) structure to satisfy the terminal boundary condition.
Key results
- On CIFAR-10 with T=10 steps, the continuous-time RL method converges faster than DxMI and reaches consistently lower FID.
- On Stable Diffusion v1.5, the fine-tuned policy's ImageReward curves at 25, 50, and 100 sampling steps nearly overlap after 20 epochs, indicating robustness to time discretization.
- Continuous-time RL shows a more stable and higher average reward than discrete-time DDPO at the same 50 discretization steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RM(x,c) with RM(x̂θ,c) in the reward mean predictor (ablation) | Value network · Value network pretraining | MSE: 2.63 -> 0.67 | Helped |
| Replace F(x,c) with F(x̂θ,c) in the residual corrector (ablation) | Value network with denoised reward mean predictor · Value network pretraining | MSE: 0.67 -> 0.66 | Helped |
| Use c_out = sin(πt/2T) instead of 1-cos(πt/2T) (ablation) | Denoised+Orig value network · Value network pretraining | MSE: 0.67 -> 0.29 | Helped |
| Use F(x̂θ,c) instead of F(x,c) in the residual corrector (ablation) | Value network with denoised reward mean predictor · Value network pretraining | MSE: 2.51 -> 0.66 | Helped |
| Train DDPO checkpoints using 50 discretization steps and evaluate them with 100 steps instead of 50 | Stable Diffusion v1.4 fine-tuned with DDPO · Fixed prompt; ImageReward | average reward increase: 0.046 -> less than 0.016 | Hurt |
| Train with continuous-time RL rather than DDPO’s fixed 50-step discretization | Stable Diffusion v1.4 · Fixed prompt; ImageReward | average reward | Helped |
| Evaluate continuous-time RL checkpoints with 25, 50, or 100 inference steps | Stable Diffusion v1.5 fine-tuned with CTRL · ImageReward | reward mean | No clear effect |
| Use continuous-time RL rather than discrete-time RL | Stable Diffusion v1.5 fine-tuning · ImageReward | average reward | Helped |
| Use continuous-time RL rather than DxMI’s policy improvement step | Short-run DDPM with T=10 · CIFAR-10 (32×32) | FID | Helped |
Surprising findings
- Discrete-time DDPO fine-tuning improves ImageReward substantially at the training discretization (50 steps) but barely improves at 100 steps, indicating overfitting to the discretization.
Limitations
- Value network evaluation relies on Monte Carlo or martingale estimates; PDE/TD-based value learning left for future work.
- Theoretical bounds rely on assumptions such as bounded reward and q-function and bounded log-ratio of densities, which may not hold generally.
- Experiments are limited to CIFAR-10 and a single prompt/reward setting for Stable Diffusion v1.5 with ImageReward.
Experimental setup
Datasets: CIFAR-10
Benchmarks: CIFAR-10
Metrics: FID average ImageReward reward
Baselines: DDPO DxMI DRaFT AlignProp
Builds on: DDPO DxMI PPO EDM ImageReward
Closest papers by meaning
- Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control (NeurIPS 2025 · similarity 0.91)
- Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods (NeurIPS 2025 · similarity 0.90)
- Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence (ICML 2025 · similarity 0.89)
- Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control (NeurIPS 2024 · similarity 0.89)
- Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model (CVPR 2024 · similarity 0.89)
- Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.