Research map / Image generation / Text-to-image diffusion

DrPO: Drifting Preference Optimization for One-Step Generative Models

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: Preference optimization for diffusion

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DrPO, an online preference-finetuning method for one-step text-to-image generators that turns reward-ranked on-policy samples into a detached feature-space drifting update (plus a reference drift), avoiding reward backpropagation.

Problem

Preference finetuning of one-step (deterministic, single-forward-pass) text-to-image generators is hard because standard alignment methods need policy likelihoods, denoising trajectories, or differentiable reward gradients, which are costly or unavailable for large VLM or black-box rewards.

Approach

Reward-ranked candidates define positive (attractive) and negative (repulsive) feature-space samples forming a non-parametric dipole preference field; a reference drift from the frozen base generator stabilizes it; the combined field gives a detached regression target. The reward is used only for ranking, so no reward gradients are needed.

Key results

• On SDXL-Turbo (Pick-a-Pic v2), DrPO reaches PickScore 23.66 and AES 6.717 versus 22.45 and 6.059 for the base model, and beats reward-gradient-free one-step baselines DPO-one (22.77), PSO (22.56), and GRPO-one (22.50).
• DrPO reduces HPSv3 update time from 21.62s to 6.17s, a 3.51x speedup over reward-gradient training under matched effective batch size.
• On SD-Turbo, DrPO raises PickScore from 21.88 to 23.49 and AES from 6.054 to 6.485 without reward backpropagation.

Surprising findings

• Raw latent features degrade performance sharply (PickScore 20.52 vs 21.88 base), showing feature choice is critical for the drift.
• Larger velocity scale does not necessarily improve performance; beta=3000 was best and larger values degraded results.

Limitations

• Inherits biases and blind spots of the target reward used for ranking.
• Still requires candidate generation, reward scoring, and feature extraction; compute advantage largest when reward backprop is expensive.
• Offline variant is only a naive extension and underperforms in principle due to off-policy, sparse pairs.

Experimental setup

Datasets: Pick-a-Pic v2 GenEval Parti-Prompts

Benchmarks: Pick-a-Pic v2 test Parti-Prompts GenEval HPSv3

Metrics: PickScore AES ImageReward HPSv2 HPSv3 CLIP GenEval score FID pairwise win rate

Baselines: DRaFT VGG-Flow DPO-one-step PSO GRPO-one-step SDXL-DPO SDXL-Turbo base SD-Turbo base

Builds on: Drifting models DRaFT VGG-Flow DPO GRPO PSO

Closest papers by meaning

• Curriculum Direct Preference Optimization for Diffusion and Consistency Models (CVPR 2025 · similarity 0.89)
• Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization (CVPR 2025 · similarity 0.88)
• Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences (ICML 2025 · similarity 0.88)
• InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment (CVPR 2025 · similarity 0.88)
• Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback (NeurIPS 2025 · similarity 0.88)
• VideoDPO: Omni-Preference Alignment for Video Diffusion Generation (CVPR 2025 · similarity 0.87)