Research map / Image generation / Autoregressive and latent generation

Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization

NeurIPS 2025 · Image generation › Autoregressive and latent generation · approach: Latent diffusion with learned tokenizers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Repurposes a pre-trained diffusion model's U-Net and text encoder as a noise-aware latent reward model (LRM) and uses it for step-level preference optimization (LPO) entirely in noisy latent space, with multi-preference consistent data filtering.

Problem

Step-level preference optimization for diffusion models relies on pixel-level VLM reward models, which need costly denoising and VAE decoding at each timestep, produce blurry predictions at high noise, and are timestep-insensitive.

Approach

Use the pre-trained diffusion model (U-Net features plus text encoder) as a reward model that scores noisy latents at any timestep, with a Visual Feature Enhancement module using classifier-free-guidance-style features and a Multi-Preference Consistent Filtering strategy on Pick-a-Pic; the same latent reward model then drives step-level preference optimization across all timesteps without decoding

Key results

• On SD1.5, LPO reaches PickScore 21.69, ImageReward 0.6588 and GenEval 48.39 (20 steps), above SPO (21.22, 0.1678, 40.46) and Diffusion-DPO (20.99, 0.3020, 43.79).
• LPO on SD1.5 needs 23 A100 hours versus 240 for Diffusion-DPO and 80 for SPO, a 10x speedup over Diffusion-DPO and 3.5x over SPO.
• On SDXL, LPO attains GenEval 59.27 and PickScore 22.86, slightly surpassing InterComp, which was trained on a higher-quality internal dataset.

Measured effects reported in the paper

Surprising findings

• LRM predicts preferences at very large timesteps [750, 950] nearly as well as across the full range, while SPO degrades when trained up to t=950.
• Adding slight noise (t=200) to latents improves LRM's preference accuracy over t=0 on Pick-a-Pic.

Limitations

• Pick-a-Pic mainly contains low-quality SD1.5 and SDXL images, which may limit generalization.
• MPCF depends on three external reward models whose biases may be inherited by the filtered data.
• The LRM must share the same VAE as the diffusion model being optimized, restricting heterogeneous use.

Experimental setup

Datasets: Pick-a-Pic v1 Pick-a-Pic v2 HPDv2 HPSv2 benchmark DPG-Bench

Benchmarks: Pick-a-Pic validation unique GenEval T2I-CompBench++ HPSv2 benchmark DPG-Bench

Metrics: PickScore ImageReward HPSv2 HPSv2.1 Aesthetic Score GenEval T2I-CompBench++ subscores Aes-Corr CLIP-Corr training time (A100 hours)

Baselines: DDPO D3PO Diffusion-DPO MaPO SPO SePPO InterComp

Builds on: Diffusion-DPO SPO DDPO CLIP classifier-free guidance

Closest papers by meaning

• Aligning Diffusion Models by Optimizing Human Utility (NeurIPS 2024 · similarity 0.92)
• Personalized Preference Fine-tuning of Diffusion Models (CVPR 2025 · similarity 0.91)
• Calibrated Multi-Preference Optimization for Aligning Diffusion Models (CVPR 2025 · similarity 0.91)
• Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback (NeurIPS 2025 · similarity 0.91)
• Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models (ICLR 2025 · similarity 0.91)
• Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · similarity 0.91)