Research map / Video / Video generation
InstructVideo: Instructing Video Diffusion Models with Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes the text-to-video diffusion model ModelScopeT2V with reward feedback from the image reward model HPSv2, using partial DDIM sampling from a noised video (reward fine-tuning as editing) plus segmental sparse reward and temporally attenuated weighting.
Problem
Text-to-video diffusion models trained on web-scale data produce visually unappealing videos misaligned with prompts, and aligning them with human preferences via reward fine-tuning is computationally expensive due to full DDIM sampling chains and the lack of video reward models.
Approach
Recast reward fine-tuning as editing: corrupt a training video to an intermediate noise level and run only a fraction of the DDIM chain, then apply off-the-shelf image reward models to sparsely sampled frames per segment (SegVR), weighted by a temporally attenuated coefficient that emphasizes central frames (TAR).
Key results
- InstructVideo raises the in-domain HPSv2 reward from 0.2542 (ModelScopeT2V, 50 DDIM steps) to 0.2717 on evaluation prompts, with 20-step inference.
- In a user study over 300 prompts, InstructVideo was preferred over ModelScopeT2V for quality 78.7% vs 8.1% (tie 13.2%) and for alignment 57.1% vs 13.8% (tie 29.1%).
- Compared with DRaFT on the same reward signals, InstructVideo was preferred for quality 75.7% vs 12.7% in the user study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune ModelScopeT2V with InstructVideo reward editing | ModelScopeT2V · In-domain animal prompts | HPSv2 score: 0.2506 ± 0.0155 -> 0.2717 ± 0.0137 | Helped |
| Fine-tune ModelScopeT2V with InstructVideo reward editing | ModelScopeT2V · New animal prompts | HPSv2 score: 0.2502 ± 0.0138 -> 0.2645 ± 0.0125 | Helped |
| Fine-tune ModelScopeT2V with InstructVideo reward editing | ModelScopeT2V · Non-animals | HPSv2 score: 0.2557 ± 0.0177 -> 0.2682 ± 0.0202 | Helped |
| Add Segmental Video Reward (SegVR) instead of dense frame reward signals (ablation) | InstructVideo · Video generation | temporal modeling capability | Helped |
| Add Temporally Attenuated Reward (TAR) instead of uniform frame weighting (ablation) | InstructVideo · Video generation | temporal modeling capability | Helped |
| Vary the editing noise level τ; τ = 0.6 balances editing space and sampling-chain cost (ablation) | InstructVideo · Reward fine-tuning evaluation prompts | HPSv2 score | Mixed |
| Vary TAR attenuation coefficient λtar; λtar = 1.0 is selected for stable fine-tuning (ablation) | InstructVideo · Reward fine-tuning evaluation prompts | HPSv2 score | Mixed |
Surprising findings
- Fine-tuned videos exceed the reward scores of the fine-tuning data itself, suggesting data quality does not cap the fine-tuned quality.
- Watermarks present in WebVid10M gradually fade during fine-tuning even though no filtering was applied.
Limitations
- Relies on a single image reward model (HPSv2) without a dedicated video reward model.
- Evaluation focuses on animal prompts with a 45-species fine-tuning set; generalization shown mainly on limited new prompts.
- Over-optimization degrades visual quality beyond about 20k steps, requiring careful stopping.
Experimental setup
Datasets: WebVid10M
Benchmarks: MSR-VTT
Metrics: HPSv2 reward score user preference (quality and alignment)
Baselines: ModelScopeT2V DDPO RWR DRaFT AlignProp
Builds on: DRaFT DDPO ModelScopeT2V HPSv2 LoRA DDIM
Closest papers by meaning
- Align-A-Video: Deterministic Reward Tuning of Image Diffusion Models for Consistent Video Editing (CVPR 2025 · similarity 0.90)
- Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuning (NeurIPS 2026 · similarity 0.89)
- Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generation (NeurIPS 2025 · similarity 0.89)
- David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.89)
- Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model (CVPR 2024 · similarity 0.88)
- Improving Video Generation with Human Feedback (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.