Research map / Image generation / Text-to-image diffusion

Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: Inference-time T2I improvement · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Diff-Instruct with Diffused Reward (DIDR), which diffuses the RLHF-optimal reward-tilted clean-image density through the reference process and trains one-step generators via a Diffused Reward Score approximated by a differentiable short-step denoising proxy, without image data.

Problem

One-step text-to-image generators aligned with human preferences via RL suffer a mismatch between terminal (clean-image) reward optimization and trajectory-level KL regularization, causing over-tilting toward high-reward outputs ("terminal reward domination").

Approach

Lift the KL-regularized RLHF optimum q*(x0) ∝ q0·exp(r/τ) to all noise levels via the forward process and minimize Integral KL against these reward-tilted marginals; the target score decomposes into the reference score plus a Diffused Reward Score, estimated by Diffused Reward Proxy using differentiable short denoising chains from the frozen reference model.

Key results

• DIDR_longer reaches PickScore 23.9 and ImageReward 1.10 on one-step SDXL, up from 23.1 and 1.01 for Diff-Instruct*.
• Standard DIDR (PickScore 23.5, FID 18.8) Pareto-dominates one-step baselines on PickScore vs FID and reports DPG-Bench 75.06 and GenEval 0.579 among one-step SDXL methods.
• On the 6B Z-Image backbone, one-step zimage-DIDR surpasses the 50-step Z-Image base on all preference metrics (ImageReward 1.08 vs 0.99).

Surprising findings

• Z-Image-DIDR trails the 8-step Z-Image-Turbo on PickScore, HPSv2.1 and GenEval despite leading on ImageReward and other metrics.
• Lowering the temperature tau improves PickScore but increases FID, so the preference-fidelity trade-off is monotone across the tested range.

Limitations

• DRP propagates gradients through the reward model, which may amplify reward-model bias or enable reward hacking.
• Each generator update needs K×S differentiable reference denoising steps, increasing training cost.
• Counting, fine-grained structure, and complex multi-object scenes remain challenging for single-step generation.

Experimental setup

Datasets: LAION-Aesthetic-6.25+ Pick-a-Pic MSCOCO-2017 validation

Benchmarks: PickScore ImageReward HPSv2.1 Aesthetic Score CLIPScore DPG-Bench GenEval FID

Metrics: PickScore ImageReward HPSv2.1 Aesthetic Score CLIPScore DPG-Bench GenEval FID

Baselines: Diff-Instruct Diff-Instruct++ Diff-Instruct* DMD2 SDXL-DPO SDXL SD3.5-large FLUX-dev Z-Image Z-Image-Turbo

Builds on: Diff-Instruct Diff-Instruct* Diff-Instruct++ DMD2 classifier guidance

Closest papers by meaning

• Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generation (NeurIPS 2025 · similarity 0.91)
• Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.91)
• David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.91)
• AdaDiff: Adaptive Step Selection for Fast Diffusion Models (AAAI 2025 · similarity 0.89)
• GARDO: Reinforcing Diffusion Models without Reward Hacking (NeurIPS 2026 · similarity 0.88)
• TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment (NeurIPS 2026 · similarity 0.88)