Research map / Image generation / Text-to-image diffusion

Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuning

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: Preference optimization for diffusion · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Diffusion-DRF, which fine-tunes a video diffusion model by backpropagating through a frozen off-the-shelf VLM's Yes/No and free-form explanation log-likelihoods on decomposed prompt questions, avoiding reward-model training and preference data.

Problem

Scalar reward models for video diffusion alignment require preference data and extra training, and give coarse global supervision that leads to reward hacking and unstable optimization.

Approach

Use a frozen VLM as a training-free critic: decompose each prompt into multi-dimensional Yes/No questions (text-video alignment, physical fidelity, visual quality) with free-form explanations, and backpropagate the summed token log-likelihoods of these responses through truncated denoising steps into the DiT.

Key results

• Diffusion-DRF (7B) reaches 55.38 overall on VBench-2.0 vs 52.99 for Wan2.1-1.3B-T2V base and 50.64 for Flow-GRPO (about 4.74 points over Flow-GRPO).
• Diffusion-DRF (1B) and (3B) reach 53.10 and 53.72 overall, showing gains grow with VLM scale.
• Human evaluation on 200 VideoGen-Eval prompts: 29% win, 54% tie, 17% lose versus base model on text-video alignment.

Surprising findings

• A simple single-question VLM yes/no reward (Vanilla-DRF) already outperforms VideoAlign on VBench-2.0 overall (53.70 vs 53.55).
• Chain-of-thought explanations improve early training but cause earlier collapse (overall drops from 54.25 at 1K steps to 51.89 at 3K steps).

Limitations

• Memory-intensive: back-propagating VLM feedback through the diffusion chain needs about 75 GB VRAM and heavy engineering.
• Evaluation of pairwise VideoGen-Eval relies on VideoAlign, which is also a baseline reward model.
• Reward hacking and stability claims rest mainly on training curves and a single base model (Wan2.1-1.3B) plus CogVideoX generalization.

Experimental setup

Datasets: OpenVid-1M VBench-2.0 VideoGen-Eval

Benchmarks: VBench-2.0 VideoGen-Eval

Metrics: VBench-2.0 overall score and sub-dimensions VideoAlign pairwise win/tie/loss rates human win/tie/lose on text-video alignment

Baselines: Flow-DPO Flow-GRPO PickScore VideoAlign Vanilla-DRF VADER InstructVideo DenseDPO

Builds on: Flow-GRPO VideoAlign Wan2.1 DreamBooth

Closest papers by meaning

• InstructVideo: Instructing Video Diffusion Models with Human Feedback (CVPR 2024 · similarity 0.89)
• Personalized Preference Fine-tuning of Diffusion Models (CVPR 2025 · similarity 0.87)
• Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.87)
• Align-A-Video: Deterministic Reward Tuning of Image Diffusion Models for Consistent Video Editing (CVPR 2025 · similarity 0.87)
• PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models (CVPR 2024 · similarity 0.87)
• Directly Fine-Tuning Diffusion Models on Differentiable Rewards (ICLR 2024 · similarity 0.87)