Research map / Video / Video generation
T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a video consistency model from VideoCrafter2 and ModelScopeT2V while adding image-text and video-text reward feedback to the consistency distillation loss, using LoRA, yielding 4-step T2V generation that tops VBench.
Problem
Video diffusion models are slow to sample, and existing video consistency models inherit the teacher's quality limits and degrade with fewer steps; prior reward-finetuning methods are memory-expensive over iterative sampling and use only image-level rewards.
Approach
Integrate differentiable rewards from an image-text RM (HPSv2.1) and a video-text RM (InternVid2 S2 or ViCLIP) directly into consistency distillation by backpropagating through the single-step consistency output, avoiding full iterative sampling gradients.
Key results
- 4-step T2V-Turbo (VC2) reaches VBench Total Score 81.01, above Gen-2 (80.58), Pika (80.40), and its teacher VideoCrafter2 (80.44).
- 4-step T2V-Turbo (MS) reaches Total Score 80.62 versus its VCM baseline at 75.84.
- Human evaluation on 700 EvalCrafter prompts finds 4-step T2V-Turbo generations preferred over the 50-step DDIM teacher samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add mixed image-text and video-text reward feedback during consistency distillation | VideoCrafter2 distilled as T2V-Turbo (VC2) · VBench | Total Score: 73.97 -> 81.01 | Helped |
| Add mixed image-text and video-text reward feedback during consistency distillation | ModelScopeT2V distilled as T2V-Turbo (MS) · VBench | Total Score: 75.84 -> 80.62 | Helped |
| Add image-text reward feedback (ablation) | VCM (MS) · VBench | Total Score: 75.84 -> 79.51 | Helped |
| Add video-text reward feedback (ablation) | VCM (MS) · VBench | Semantic Score: 63.98 -> 71.35 | Helped |
| Add image-text reward feedback (ablation) | VCM (VC2) · VBench | Total Score: 73.97 -> 80.42 | Helped |
| Add video-text reward feedback (ablation) | VCM (VC2) · VBench | Semantic Score: 55.66 -> 67.55 | Helped |
| Add video-text reward feedback alongside image-text reward feedback (ablation) | VCM (MS) + image-text reward · VBench | Semantic Score: 70.32 -> 74.47 | Helped |
| Add video-text reward feedback alongside image-text reward feedback (ablation) | VCM (VC2) + image-text reward · VBench | Semantic Score: 71.70 -> 74.76 | Helped |
| Replace InternVid2 S2 with ViCLIP as the video-text reward model (ablation) | T2V-Turbo (MS) · VBench | Semantic Score: 70.41 -> 74.47 | Helped |
| Replace InternVid2 S2 with ViCLIP as the video-text reward model (ablation) | T2V-Turbo (VC2) · VBench | Semantic Score: 74.76 -> 73.52 | Hurt |
| Use 8 inference steps instead of 4 | T2V-Turbo · EvalCrafter human evaluation | Human preference | Helped |
| Use HPSv2.1, PickScore, or ImageReward as image-text reward feedback (ablation) | VCM (VC2) · Qualitative video comparison | Video quality | Helped |
Surprising findings
- Video-text reward from ViCLIP or InternVid2 S2 can hurt specific VBench dimensions such as dynamic degree (e.g., VCM(VC2)+R_vid drops to 25.00), while improving semantic score.
- Image-text reward alone (VCM + R_img) gives most of the quality gain, with the video-text reward mainly improving semantic score.
Limitations
- No open-sourced video-text reward model trained on human preferences is available, so general video foundation models (ViCLIP, InternVid2 S2) are used as the video-text reward.
- Results depend on the choice of video-text reward: ViCLIP vs InternVid2 S2 produce different trade-offs across VBench dimensions and for ModelScopeT2V.
- Human evaluation covers only 700 prompts with AMT annotators.
Experimental setup
Datasets: WebVid-10M EvalCrafter prompts (700, for human evaluation)
Benchmarks: VBench EvalCrafter
Metrics: VBench Total Score VBench Quality Score VBench Semantic Score human preference win rate (visual appeal, text alignment, ov
Baselines: VCM (VC2) VCM (MS) Gen-2 Pika VideoCrafter2 ModelScopeT2V VideoCrafter1 Show-1 LaVie InstructVideo
Builds on: latent consistency distillation InstructVideo VideoCrafter2 VBench HPSv2
Closest papers by meaning
- T2V-Turbo-v2: Enhancing Video Model Post-Training through Data, Reward, and Conditional Guidance Design (ICLR 2025 · similarity 0.92)
- OSV: One Step is Enough for High-Quality Image to Video Generation (CVPR 2025 · similarity 0.88)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.88)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.88)
- Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis (CVPR 2025 · similarity 0.88)
- VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.