Research map / Reinforcement learning / RL from feedback

Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model

CVPR 2024 · Reinforcement learning › RL from feedback · approach: Theory of RLHF · 49 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes D3PO, which adapts DPO to the multi-step denoising MDP of diffusion models so they can be fine-tuned directly from human preference comparisons without training a reward model.

Problem

RLHF for diffusion models requires training a reward model, which is costly in data, architecture, and tuning; directly applying DPO to diffusion models is infeasible due to GPU memory needed to store gradients across many latent denoising steps.

Approach

Reformulate the denoising process as a multi-step MDP, show via a Q-function/Bradley-Terry argument that the DPO objective implicitly learns the optimal reward, and apply the DPO loss to every state-action pair of sub-trajectories, using the final image's human preference for all denoising steps.

Key results

• On quantitative objectives, D3PO using only relative preferences achieves results nearly on par with DDPO, DPOK and Reward Weighted trained with ground-truth rewards.
• On prompt-image alignment, D3PO scores CLIP 31.9, BLIP 2.06, ImageReward 0.27 versus 30.7/1.95/0.04 for the pre-trained model and 31.5/2.01/0.17 for Reward Weighted.
• In human evaluation over 300 prompts, D3PO was preferred in 55.7% of cases versus 14.7% for the pre-trained model and 18.7% for Reward Weighted.

Measured effects reported in the paper

Surprising findings

• Using only relative sizes of rewards as preferences nearly matches methods trained with ground-truth rewards.

Limitations

• Human feedback is needed for each comparison, which is costly to scale.
• Preferences are only given on final images but applied to all denoising steps, a sub-trajectory assumption that may be imprecise.
• Human evaluation relies on few annotators (two for safety), limiting reliability.

Experimental setup

Datasets: LAION aesthetics predictor training data

Benchmarks: CLIP score BLIP score ImageReward score human preference evaluation on 300 prompts

Metrics: image size reward LAION aesthetic score normalcy rate of hand images deformity rate of anime characters safety rate CLIP score BLIP score ImageReward score human preference percentage

Baselines: DDPO DPOK Reward Weighted ReFL ImageReward-based fine-tuning

Builds on: DPO DDPO Reward Weighted regression LoRA

Closest papers by meaning

• Training Diffusion Models with Reinforcement Learning (ICLR 2024 · similarity 0.93)
• Directly Fine-Tuning Diffusion Models on Differentiable Rewards (ICLR 2024 · similarity 0.92)
• Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback (NeurIPS 2025 · similarity 0.92)
• Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · similarity 0.90)
• David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.90)
• Self-Play Fine-tuning of Diffusion Models for Text-to-image Generation (NeurIPS 2024 · similarity 0.89)