Research map / Reinforcement learning / RL from feedback
PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PRDP, a supervised reward-difference regression objective with online sampling and proximal (PPO-style) clipping that enables stable black-box reward finetuning of Stable Diffusion v1.4 on over 100K prompts.
Problem
RL-based reward finetuning of diffusion models (e.g., DDPO) is unstable at large prompt scale, so it fails to generalize to complex, unseen prompts.
Approach
Derive from the KL-regularized RLHF objective a supervised MSE loss that trains the diffusion model to predict reward differences between denoising-trajectory log-ratio pairs of images for the same prompt; combine it with online sampling and per-step clipped proximal updates, avoiding policy gradients.
Key results
- In large-scale HPSv2 training, PRDP raises average HPSv2 on unseen HPDv2 photo prompts from 0.2750 (SD v1.4) to 0.3159, while DDPO drops to 0.2093.
- In small-scale training (45 prompts), PRDP achieves HPSv2 0.3471 vs DDPO 0.3398 and SD v1.4 0.2855.
- Online optimization matches offline performance in about 10 epochs and keeps improving; DDPO without KL regularization reward-hacks the aesthetic predictor while PRDP with beta=10 preserves text-image alignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PRDP reward finetuning instead of Stable Diffusion v1.4 | Stable Diffusion v1.4 · 45 animal prompts | HPSv2 reward score: 0.2855 -> 0.3471 | Helped |
| Use PRDP reward finetuning instead of Stable Diffusion v1.4 | Stable Diffusion v1.4 · 45 animal prompts | PickScore reward score: 0.2179 -> 0.2700 | Helped |
| Use PRDP rather than DDPO | Stable Diffusion v1.4 · HPDv2 training set prompts | HPSv2 reward score: 0.2464 -> 0.3175 | Helped |
| Use PRDP rather than DDPO | Stable Diffusion v1.4 · Pick-a-Pic v1 test set prompts | HPSv2 reward score: 0.2501 -> 0.3050 | Helped |
| Use PRDP rather than DDPO | Stable Diffusion v1.4 · HPDv2 animation | HPSv2 reward score: 0.2673 -> 0.3223 | Helped |
| Use PRDP rather than DDPO | Stable Diffusion v1.4 · HPDv2 photo | HPSv2 reward score: 0.2093 -> 0.3159 | Helped |
| Use PRDP rather than DDPO | Stable Diffusion v1.4 · HPDv2 training set prompts | PickScore reward score: 0.2032 -> 0.2424 | Helped |
| Use PRDP rather than DDPO | Stable Diffusion v1.4 · Pick-a-Pic v1 test set prompts | PickScore reward score: 0.1992 -> 0.2344 | Helped |
| Use PRDP rather than DDPO | Stable Diffusion v1.4 · HPDv2 concept art | PickScore reward score: 0.2125 -> 0.2441 | Helped |
| Add proximal updates (ablation) | PRDP · 45 animal prompts | training stability | Helped |
| Use online rather than offline optimization (ablation) | PRDP · 45 animal prompts | HPSv2 reward score | Helped |
| Use online rather than offline optimization (ablation) | PRDP · 45 animal prompts | PickScore reward score | Helped |
Surprising findings
- DDPO fails at large scale, producing results with lower reward than the base Stable Diffusion model.
- DDPO without KL regularization completely ignores text prompts when optimizing the aesthetic score.
Limitations
- Comparison to DPOK is omitted; only DDPO is compared as the RL baseline.
- Experiments are limited to Stable Diffusion v1.4 and a few reward models, with evaluation using the same reward models used for training.
- Hyperparameters such as clipping range and beta are tuned per setup, with limited analysis.
Experimental setup
Datasets: HPDv2 Pick-a-Pic v1
Benchmarks: HPDv2 (animation, concept art, painting, photo) Pick-a-Pic v1 test prompts
Metrics: HPSv2 reward score PickScore reward score aesthetic score
Baselines: DDPO Stable Diffusion v1.4 offline optimization
Builds on: DPO PPO DDPO DPOK DRaFT
Closest papers by meaning
- Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.89)
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.89)
- Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model (CVPR 2024 · similarity 0.89)
- Personalized Preference Fine-tuning of Diffusion Models (CVPR 2025 · similarity 0.89)
- Directly Fine-Tuning Diffusion Models on Differentiable Rewards (ICLR 2024 · similarity 0.88)
- Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.