Research map / Video / Video-language understanding
Boosting Text-to-Video Generative Model with MLLMs Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses GPT-4V to annotate 135K video preference labels (VIDEOPREFER), trains a temporal-modeling reward model VIDEORM on top of HPS v2, and fine-tunes ModelScopeT2V with a truncated-backprop reward method DRaFT-V.
Problem
Text-to-video generators produce visually unappealing, prompt-misaligned videos, but no large-scale video preference data or video-specific reward model exists for RLHF-style alignment.
Approach
Use GPT-4V as a preference annotator for video prompt-following and quality, build a video preference dataset from it, and train a video reward model by adding temporal shift and a temporal transformer to the image reward model HPS v2, then fine-tune T2V models with truncated reward backprop (DRaFT-V).
Key results
- VIDEORM achieves 67.5 average pairwise preference accuracy across TVGE, VBench and T2VQA-DB, versus 59.9 for PickScore and 59.3 for HPS v2.
- GPT-4V has the highest agreement with human labels on TVGE among tested MLLMs, with 69.65% accuracy and Kendall tau 0.360.
- Human study shows DRaFT-V fine-tuned ModelScopeT2V is preferred over baseline in win-rate comparisons; VIDEORM inference takes 1.3s vs 5s for HPS v2 per 8-frame video.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VIDEORM instead of HPS v2 for preference scoring | video preference prediction · TVGE | preference accuracy: 69.5 -> 73.7 | Helped |
| use VIDEORM instead of HPS v2 for preference scoring | video preference prediction · VBench | preference accuracy: 55.7 -> 63.5 | Helped |
| use VIDEORM instead of PickScore for preference scoring | video preference prediction · T2VQA-DB | preference accuracy: 61.2 -> 65.4 | Helped |
| add VIDEOPREFER training data to open-source preference data (ablation) | VIDEORM · TVGE | preference accuracy: 72.8 -> 73.7 | Helped |
| add VIDEOPREFER training data to open-source preference data (ablation) | VIDEORM · VBench | preference accuracy: 60.2 -> 63.5 | Helped |
| add VIDEOPREFER training data to open-source preference data (ablation) | VIDEORM · T2VQA-DB | preference accuracy: 64.3 -> 65.4 | Helped |
| replace the temporal transformer with Conv1D (ablation) | VIDEORM · TVGE | preference accuracy: 73.7 -> 70.1 | Hurt |
| replace the temporal transformer with LSTM (ablation) | VIDEORM · TVGE | preference accuracy: 73.7 -> 73.9 | No clear effect |
| initialize with PickScore instead of HPS v2 (ablation) | VIDEORM · VBench | preference accuracy: 63.5 -> 64.2 | Helped |
| initialize with ImageReward instead of HPS v2 (ablation) | VIDEORM · TVGE | preference accuracy: 73.7 -> 72.4 | Hurt |
| increase the VIDEOPREFER training dataset size (ablation) | VIDEORM · TVGE, VBench, and T2VQA-DB | preference accuracy | Helped |
| use fewer input video frames (ablation) | VIDEORM · TVGE | preference accuracy | Hurt |
Surprising findings
- An image-domain-initialized reward model trained only on GPT-4V annotations (VIDEORM-V) already outperforms all image reward baselines on out-of-distribution benchmarks.
- Replacing the temporal transformer with LSTM gives comparable performance, while Conv1D causes a significant drop.
Limitations
- Annotations rely on GPT-4V with only ~70% accuracy relative to ground truth and limited to 10 frames; the authors' accuracy analysis shows sensitivity to frame count and temperature.
- Fine-tuning experiments use only 2K prompt-video pairs and 400 steps on a 256x256 16-frame base model.
- No error bars or statistical significance reported due to compute limits.
Experimental setup
Datasets: VIDEOPREFER TVGE VBench T2VQA-DB VidProM ActivityNet MSR-VTT WebVid10M
Benchmarks: TVGE VBench T2VQA-DB
Metrics: pairwise preference accuracy Spearman correlation Kendall tau win rate best-of-n win rate
Baselines: CLIP ViT-H/14 ImageReward PickScore HPS v2 InstructVideo DRaFT-H Aesthetic classifier
Builds on: HPS v2 DRaFT InstructVideo ModelScopeT2V
Closest papers by meaning
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better (NeurIPS 2024 · similarity 0.90)
- Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation (CVPR 2025 · similarity 0.89)
- Improving Video Generation with Human Feedback (NeurIPS 2025 · similarity 0.88)
- Preference Adaptive and Sequential Text-to-Image Generation (ICML 2025 · similarity 0.88)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.