Research map / Video / Video generation
Improving Video Generation with Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 182k-triplet human preference dataset for modern text-to-video models, trains a multi-dimensional VLM reward model (VideoReward) with BT-with-ties and token positioning, and proposes Flow-DPO, Flow-RWR, and Flow-NRG reward-guidance for rectified-flow video alignment.
Problem
Video generation with rectified flow still shows unsmooth motion and text-video misalignment, and existing video reward datasets and alignment methods (designed for older diffusion models) do not transfer well to modern flow-based T2V models.
Approach
Derives a flow-matching DPO loss whose KL weight is constant over timesteps (dropping the (1-t)^2 factor of Diffusion-DPO), pairs it with a BT-with-ties multi-dimensional VLM reward model using context-agnostic dimension tokens, and applies reward guidance on noisy latents via a time-dependent latent reward model.
Key results
- VideoReward reaches 61.26% overall accuracy with ties and 73.59% without ties on VideoGen-RewardBench, above VisionReward (56.77% / 67.59%).
- Flow-DPO with constant beta achieves TA win rate 73.24 on TA-Hard multi-dimensional alignment and 84.51 TA-Hard single-dimension TA, versus 38.03 and 71.83 for the timestep-dependent beta variant.
- In single-dimension TA alignment, Flow-DPO reaches VBench total 83.38 vs pretrained 83.19 while Flow-RWR and SFT fall below pretrained on VBench total.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoReward instead of VisionReward | Video reward modeling · VideoGen-RewardBench | overall accuracy w/ ties: 56.77 -> 61.26 | Helped |
| Train with BTT instead of pointwise regression (ablation) | Qwen2-VL-2B reward model · VideoGen-RewardBench | overall accuracy w/ ties: 58.39 -> 61.50 | Helped |
| Add explicit tie handling with BTT instead of BT (ablation) | Qwen2-VL-2B reward model · VideoGen-RewardBench | overall accuracy w/ ties: 61.22 -> 61.50 | Helped |
| Add separate tokens for each reward attribute (ablation) | BTT reward model · VideoGen-RewardBench | MQ accuracy w/ ties: 64.64 -> 66.03 | Helped |
| Use constant β instead of timestep-dependent β (ablation) | Flow-DPO · VideoGen-Eval | TA win rate: 28.14 -> 68.26 | Helped |
| Use constant β instead of timestep-dependent β (ablation) | Flow-DPO · VBench | Total: 80.90 -> 83.41 | Helped |
| Replace SFT with Flow-DPO using constant β | Internal flow video model · VBench | Total: 82.31 -> 83.41 | Helped |
| Replace Flow-RWR with Flow-DPO using constant β | Internal flow video model · VBench | Total: 82.27 -> 83.41 | Helped |
| Set Flow-NRG weights to emphasize VQ and MQ and omit TA | Flow-NRG · VideoGen-Eval | MQ win rate: 46.48 -> 93.23 | Helped |
| Set Flow-NRG weights to emphasize VQ and MQ and omit TA | Flow-NRG · VideoGen-Eval | TA win rate: 70.42 -> 26.65 | Hurt |
| Train the reward model on noisy latents rather than clean latents (ablation) | Flow-NRG reward guidance · TA-Hard | MQ win rate: 38.6 -> 74.6 | Helped |
| Use the timestep-dependent β schedule (ablation) | Flow-DPO · VideoGen-Eval | TA win rate: 50.0 -> 28.14 | Hurt |
Surprising findings
- Diffusion-DPO-style timestep-dependent KL weighting (beta(1-t)^2) causes reward hacking and degrades text alignment in rectified flow, while a constant beta works better.
- Multi-dimensional Flow-DPO with timestep-dependent beta underperforms the pretrained model on TA despite high VQ/MQ win rates.
Limitations
- Excessive Flow-DPO training degrades video quality, mitigated with LoRA.
- Reward model remains susceptible to reward hacking since the reward is differentiable.
- Validated only on text-to-video, not image-to-video or other conditional tasks.
Experimental setup
Datasets: VideoGen-Eval GenAI-Bench VBench TA-Hard
Benchmarks: GenAI-Bench VideoGen-RewardBench VBench VideoGen-Eval TA-Hard
Metrics: pairwise accuracy (w/ ties, w/o ties) win rate VBench total/quality/semantic score human preference win rate
Baselines: VideoScore LiFT VisionReward SFT Flow-RWR Flow-DPO with timestep-dependent beta
Builds on: Diffusion-DPO Reward-weighted regression Rectified Flow Bradley-Terry model with ties Guided flows
Closest papers by meaning
- Boosting Text-to-Video Generative Model with MLLMs Feedback (NeurIPS 2024 · similarity 0.88)
- InstructVideo: Instructing Video Diffusion Models with Human Feedback (CVPR 2024 · similarity 0.88)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.88)
- VBench: Comprehensive Benchmark Suite for Video Generative Models (CVPR 2024 · similarity 0.87)
- Align-A-Video: Deterministic Reward Tuning of Image Diffusion Models for Consistent Video Editing (CVPR 2025 · similarity 0.87)
- Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.