Research map / Image generation / Text-to-image diffusion

Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models

ICLR 2025 · Image generation › Text-to-image diffusion · approach: Preference optimization for diffusion · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Negative Preference Optimization (NPO), which trains a negative-preference model by reversing preference pairs (or negating reward) and uses it as the unconditional branch of classifier-free guidance to improve preference alignment of diffusion models.

Problem

Preference alignment methods for diffusion models only optimize the conditional branch and ignore the unconditional/negative-conditional outputs used in classifier-free guidance, limiting their ability to avoid undesirable generations.

Approach

Train a second weight offset on reversed preference pairs (equivalently, negated reward) so it models negative preferences, then use it as the negative-conditional predictor in CFG alongside the preference-optimized model; this needs no new datasets or training strategies and can be plugged in at inference.

Key results

• On SD1.5, Diff.-DPO + NPO raises HPSv2 from 25.05 to 27.60 and ImageReward from 0.1115 to 0.3101 (reg=500), and DreamShaper + NPO reaches HPSv2 28.31 vs 27.97.
• On SDXL, Diff.-SPO + NPO achieves PickScore 23.08 vs 22.97 and ImageReward 1.047 vs 1.032, with Aesthetic 6.438 vs 6.348.
• On text-to-video, VADER + NPO (HPSv2) gives a 94.79% aesthetic win rate vs 55.21% for VADER on animal prompts.

Measured effects reported in the paper

Surprising findings

• Scaling DPO training to 10x iterations does not match DPO (k=1) + NPO (k=1), which outperforms the longer-trained DPO at a fraction of the cost.
• Adding mild Gaussian-blur corruptions to negative images slightly improved NPO, while text-prompt shuffling degraded it.

Limitations

• Inference needs two weight offsets, increasing storage and loading cost.
• Hyperparameters (alpha, beta, regularization) require tuning and substantially affect quality; too large deviation yields blurred images.
• Performance gains on some metrics are mixed (e.g., Diff.-SPO + NPO has slightly lower HPSv2 on SD1.5 and Aesthetic changes are small on some SDXL settings).

Experimental setup

Datasets: Pick-a-pic

Benchmarks: Pick-a-pic validation_unique / test_unique prompts animal and human prompts for text-to-video

Metrics: PickScore HPSv2 ImageReward LAION-Aesthetic win rate user preference

Baselines: DDPO D3PO Diffusion-DPO Diffusion-SPO VADER VideoCrafter2 SEG Autoguidance

Builds on: Diffusion-DPO Diffusion-SPO VADER classifier-free guidance

Closest papers by meaning

• Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback (NeurIPS 2025 · similarity 0.92)
• Aligning Diffusion Models by Optimizing Human Utility (NeurIPS 2024 · similarity 0.91)
• Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization (NeurIPS 2025 · similarity 0.91)
• Calibrated Multi-Preference Optimization for Aligning Diffusion Models (CVPR 2025 · similarity 0.91)
• Personalized Preference Fine-tuning of Diffusion Models (CVPR 2025 · similarity 0.91)
• Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences (ICML 2025 · similarity 0.91)