Research map / Video / Video generation
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a high-quality latent text-to-video diffusion model from low-quality WebVid-10M videos by fully training the video model first, then finetuning only spatial modules on synthesized Midjourney images (JDB).
Problem
Training high-quality text-to-video diffusion models without access to large-scale high-quality videos, since academic video datasets like WebVid-10M are low resolution and low quality, and models trained on them produce poor picture quality.
Approach
Analyzes spatial-temporal coupling in SD-based video models: fully trained models (all modules) have stronger coupling and tolerate parameter perturbation, so the paper disentangles motion from appearance at the data level by learning motion from low-quality videos, then directly finetuning spatial modules on high-quality synthesized images to raise picture quality and concept composition.
Key results
- On EvalCrafter, the method reaches visual quality 63.28 vs VideoCrafter1 61.64 and Show-1 52.19, with text-video alignment 64.67 and motion quality 53.95.
- Directly finetuning spatial modules (F-Spa-DIR) raises DOVER aesthetic score from 46.55 (F-base) to 82.57 while keeping motion preferred 67% over F-Spa&Temp-DIR.
- In user study, the method is preferred over AnimateDiff for visual quality 69% and over Show-1 for visual quality 82%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train spatial and temporal modules together instead of training only temporal modules (ablation) | VideoCrafter-style T2V model · perturbed T2V models | motion quality: 87% | Helped |
| Train spatial and temporal modules together instead of training only temporal modules (ablation) | VideoCrafter-style T2V model · perturbed T2V models | motion quality: 73% | Helped |
| Fine-tune spatial modules with LORA rather than temporal modules with LORA (ablation) | fully trained T2V model · perturbed T2V models | aesthetic score: 66.26 -> 77.97 | Helped |
| Directly fine-tune spatial modules rather than using LORA (ablation) | fully trained T2V model · perturbed T2V models | aesthetic score: 77.97 -> 82.57 | Helped |
| Fine-tune spatial and temporal modules together instead of spatial modules only (ablation) | fully trained T2V model · perturbed T2V models | motion quality: 67% | Hurt |
| Fine-tune spatial and temporal modules together with LORA rather than spatial modules only with LORA (ablation) | fully trained T2V model · perturbed T2V models | aesthetic score: 77.97 -> 80.44 | Helped |
| Directly fine-tune temporal modules instead of using LORA (ablation) | fully trained T2V model · perturbed T2V models | aesthetic score: 66.26 -> 82.77 | Helped |
| Use Midjourney-synthesized JDB images instead of LAION Aesthetics V2 images for spatial fine-tuning (ablation) | fully trained T2V model · perturbed T2V models | aesthetic score: 67.83 -> 82.57 | Helped |
| Use Midjourney-synthesized JDB images instead of LAION Aesthetics V2 images for spatial fine-tuning (ablation) | fully trained T2V model · perturbed T2V models | technical score: 54.26 -> 70.35 | Helped |
| Use synthetic JDB images for fine-tuning rather than web-collected LAION images (ablation) | VideoCrafter2 · video generation | concept composition | Helped |
| Use low-quality WebVid videos for motion learning and high-quality JDB images for appearance fine-tuning | VideoCrafter2 · EvalCrafter | visual quality: 61.64 -> 63.28 | Helped |
| Use low-quality WebVid videos for motion learning and high-quality JDB images for appearance fine-tuning | VideoCrafter2 · EvalCrafter | motion quality: 56.06 -> 53.95 | Hurt |
Surprising findings
- Fully trained models tolerate parameter perturbation better: partially trained models degrade in motion quickly when finetuned, while fully trained ones do not.
Limitations
- Motion quality still falls short of models trained on larger video volumes (Gen-2, Pika Labs).
- Relies on the spatial-temporal coupling observed in one architecture (VideoCrafter1-based), and motion-quality evaluation lacks a comprehensive objective metric so relies on user studies.
- Output picture quality shifts to the Midjourney style of JDB images.
Experimental setup
Datasets: WebVid-10M LAION-COCO JDB LAION-aesthetics V2
Benchmarks: EvalCrafter
Metrics: EvalCrafter visual quality text-video alignment motion quality temporal consistency DOVER aesthetic score DOVER technical score user preference
Baselines: Gen-2 Pika Labs VideoCrafter1 Show-1 AnimateDiff
Builds on: VideoCrafter1 Stable Diffusion ModelScopeT2V AnimateDiff LoRA
Closest papers by meaning
- VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide (CVPR 2025 · similarity 0.90)
- VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.89)
- Grid Diffusion Models for Text-to-Video Generation (CVPR 2024 · similarity 0.89)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.89)
- OSV: One Step is Enough for High-Quality Image to Video Generation (CVPR 2025 · similarity 0.89)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.