Research map / Video / Video generation
VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoGuide, a training-free inference-time method that interpolates denoised samples from a teacher video diffusion model into a student's early denoising steps to improve temporal consistency of text-to-video generation.
Problem
Text-to-video diffusion models trade off temporal consistency against imaging quality, and existing fixes such as FreeInit either degrade detail or add substantial inference time.
Approach
Treat video consistency as a guidance optimization problem and, during early reverse-sampling steps, interpolate the denoised latent of the sampling model with the denoised output of a guiding VDM (any pretrained model, or the model itself) obtained by a few extra denoising steps, followed by low-pass filtering; no training is needed.
Key results
- On AnimateDiff, VideoGuide with VideoCrafter2 guidance raises subject consistency from 0.9183 to 0.9614 and background consistency from 0.9437 to 0.9664, with imaging quality 0.6671 vs 0.6647 for the base model.
- VideoGuide is about 1.8-2.4x faster than FreeInit for AnimateDiff and 2.1-3.0x faster for LaVie (Table 2), with minimal extra GPU memory for self-guided use (8.35 GB same as AnimateDiff).
- Using VideoCrafter2 as guide raises LaVie's subject consistency from 0.9534 to 0.9635 while keeping imaging quality (0.6796 vs 0.6750) and reducing dynamic degree (14.69 to 12.63).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply VideoGuide with VideoCrafter2 as guide | AnimateDiff · Video generation | Subject Consistency: 0.9183 -> 0.9614 | Helped |
| Apply VideoGuide with VideoCrafter2 as guide | AnimateDiff · Video generation | Imaging Quality: 0.6647 -> 0.6671 | No clear effect |
| Apply VideoGuide with VideoCrafter2 as guide | AnimateDiff · Video generation | Dynamic Degree: 26.67 -> 16.78 | Hurt |
| Apply VideoGuide with VideoCrafter2 as guide | LaVie · Video generation | Subject Consistency: 0.9534 -> 0.9635 | Helped |
| Apply FreeInit | AnimateDiff · Video generation | Imaging Quality: 0.6647 -> 0.6173 | Hurt |
| Vary interpolation scale β from 0.9 to 0.5 (ablation) | VideoGuide (AnimateDiff) · Video generation | Subject Consistency: 0.9518 -> 0.9546 -> 0.9576 -> 0.9605 -> 0.9614 | Helped |
| Vary interpolation scale β from 0.9 to 0.5 (ablation) | VideoGuide (AnimateDiff) · Video generation | Background Consistency: 0.9599 -> 0.9609 -> 0.9628 -> 0.9649 -> 0.9664 | Helped |
| Vary interpolation step number I from 1 to 5 (ablation) | VideoGuide (AnimateDiff) · Video generation | Subject Consistency: 0.9524 -> 0.9489 -> 0.9546 -> 0.9602 -> 0.9614 | Mixed |
| Vary interpolation step number I from 1 to 5 (ablation) | VideoGuide (AnimateDiff) · Video generation | Background Consistency: 0.9618 -> 0.9588 -> 0.9612 -> 0.9645 -> 0.9664 | Mixed |
| Vary guidance sampling steps τ from 1 to 10 (ablation) | VideoGuide (AnimateDiff) · Video generation | Subject Consistency: 0.9444 -> 0.9531 -> 0.9582 -> 0.9611 -> 0.9614 | Helped |
| Vary guidance sampling steps τ from 1 to 10 (ablation) | VideoGuide (AnimateDiff) · Video generation | Background Consistency: 0.9558 -> 0.9611 -> 0.9641 -> 0.9658 -> 0.9664 | Helped |
Surprising findings
- Dynamic degree drops notably for AnimateDiff with VideoGuide (26.67 to 16.78 with VideoCrafter2 guidance), a trade-off the paper acknowledges.
- FreeInit improves consistency but degrades imaging quality (AnimateDiff MUSIQ 0.6647 to 0.6173).
Limitations
- Trade-off in dynamic degree: motion magnitude reduced relative to the base model, especially with external guidance.
- Guidance requires running an additional video diffusion model, increasing inference time and memory (15.71 GB for VC-guided vs 8.35 GB) and requires noise schedule alignment.
- Interpolation weight is constrained to 0.5 minimum to avoid distribution shift, and approximation of the teacher's PF-ODE endpoint uses multi-step sampling rather than a true consistency model.
Experimental setup
Benchmarks: VBench
Metrics: subject consistency background consistency imaging quality (MUSIQ) motion smoothness dynamic degree inference time GPU memory usage
Baselines: FreeInit AnimateDiff LaVie VideoCrafter2
Builds on: DDIM decomposed diffusion sampling FreeInit AnimateDiff score distillation sampling
Closest papers by meaning
- VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.92)
- MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance (AAAI 2024 · similarity 0.91)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.90)
- TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.90)
- Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis (CVPR 2025 · similarity 0.90)
- COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.