Research map / Reinforcement learning / RL from feedback

Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Unifies many reward-based fine-tuning methods for flow/diffusion models as regression to a value-guided score (reward score matching) and uses the framework to redesign zeroth- and first-order methods for better compute efficiency.

Problem

Reward-based fine-tuning of diffusion and flow models is fragmented across soft RL, optimal control, direct reward backpropagation, and GFlowNet-style derivations, making it hard to tell which design choices matter.

Approach

Shows that many methods (Soft RL variants, VGG-Flow, DRaFT/DRaFT-K, ReFL, Adjoint Matching) share one L2 score-matching loss toward a reference score plus value guidance; they differ mainly in the value-guidance estimator (first-order current-state/lookahead vs zeroth-order policy-gradient lookahead) and in timestep-wise effective strength. Uses this to redesign budget allocation (zeroth-order) an

Key results

• Zeroth-order redesign on flow matching reaches GenEval 0.97 with an estimated 5x training-cost reduction relative to the reported TempFlow-GRPO baseline.
• Sparse branching allocation for zeroth-order diffusion (PCPO-based) improves the reward-KL tradeoff over the PCPO baseline with uniform K_i=1.
• Replacing local Tweedie first-order gradients with terminal-image reward gradients plus reversed low-SNR downweighting yields faster reward gains on SD3.5-M and SD1.5 with competitive reward-KL tradeoff.

Surprising findings

• Shallower lookahead estimators can outperform unbiased full-rollout estimators under finite compute.
• Learned residual correction terms (g_phi) in existing first-order methods are effectively negligible.

Limitations

• Does not eliminate reward hacking and does not establish robustness to reward misspecification.
• Does not cover offline preference-based objectives such as Diffusion-DPO.
• Some GenEval baseline results are reverse-engineered from a figure in a prior paper rather than rerun.

Experimental setup

Datasets: GenEval training prompts hpd_photo_painting prompts

Benchmarks: GenEval HPSv2.1 Aesthetic Score

Metrics: GenEval score aesthetic score KL divergence transition mean drift GPU hours

Baselines: TempFlow-GRPO VGG-Flow PCPO DRaFT-K ReFL Flow-GRPO

Builds on: DDPO DPOK Flow-GRPO DanceGRPO TempFlow-GRPO VGG-Flow DRaFT ReFL Adjoint Matching Residual nabla-DB

Closest papers by meaning

• Directly Fine-Tuning Diffusion Models on Differentiable Rewards (ICLR 2024 · similarity 0.90)
• Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control (ICLR 2025 · similarity 0.90)
• Scaling Reward Modeling without Human Supervision (NeurIPS 2026 · similarity 0.89)
• The Reward Was in Your Data All Along: Correcting Flow Matching with Discriminator-Guided RL. (NeurIPS 2026 · similarity 0.89)
• Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases (ICML 2024 · similarity 0.89)
• Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Models (ICML 2025 · similarity 0.89)