Research map / Image generation / Text-to-image diffusion
MicroCinema: A Divide-and-Conquer Approach for Text-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MicroCinema splits text-to-video generation into a text-to-image stage (off-the-shelf, SDXL) and an image&text-to-video stage built on Stable Diffusion 2.1 with an Appearance Injection Network and Appearance Noise Prior, reaching zero-shot FVD 342.86 on UCF-101 and 377.40 on MSR-VTT.
Problem
Text-to-video generation struggles with appearance consistency and temporal coherence/motion quality when text is aligned directly with video, and fine-tuning T2I models for video can degrade the pretrained image capability.
Approach
Decompose text-to-video into a text-to-image keyframe stage and an image&text-to-video stage so the video model focuses on motion; inject the keyframe via a multi-scale denormalization-based AppearNet and add a scaled keyframe latent to the training noise (Appearance Noise Prior).
Key results
- Zero-shot FVD 342.86 on UCF-101 (IS 37.46), best among compared methods including those using extra data.
- Zero-shot FVD 377.40 on MSR-VTT (CLIPSIM 0.2967) using only WebVid-10M training.
- Appearance Noise Prior with lambda=0.03 and gamma=0.02 reduces UCF-101 FVD from 692 to 508 and raises IS from 18.5 to 29.6 (2K-sample ablation setting).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the two-stage MicroCinema framework instead of PYoCo | Zero-shot text-to-video generation · UCF-101 | FVD: 355.19 -> 342.86 | Helped |
| Use the two-stage MicroCinema framework instead of SimDA | Zero-shot text-to-video generation · MSR-VTT | FVD: 456.00 -> 377.40 | Helped |
| Use AppearNet instead of direct reference-latent concatenation (ablation) | MicroCinema · UCF-101 | FVD: 688.92 -> 508.56 | Helped |
| Inject AppearNet features into encoder and decoder instead of decoder only (ablation) | MicroCinema · UCF-101 | FVD: 589.59 -> 525.02 | Helped |
| Add SPADE-based injection into GroupNorm layers (ablation) | MicroCinema with encoder-decoder AppearNet injection · UCF-101 | FVD: 525.02 -> 508.56 | Helped |
| Add appearance noise prior with λ = 0.03 and γ = 0.02 (ablation) | MicroCinema · UCF-101 | FVD: 692 to 508 | Helped |
| Add appearance noise prior with λ = 0.03 and γ = 0.02 (ablation) | MicroCinema · UCF-101 | IS: 18.5 to 29.6 | Helped |
| Use a LLaVA-1.5-generated caption instead of a simple prompt | MicroCinema with SDXL first-stage image generation · UCF-101 | FVD: 374.05 -> 336.40 | Helped |
| Use SDXL instead of SD-2.1 for first-stage image generation | MicroCinema with LLaVA-1.5 prompts · UCF-101 | FVD: 412.53 -> 336.40 | Helped |
Surprising findings
- Make-A-Video-style two-stage approach is noted as similar, but the authors report that prompt quality (LLaVA-1.5 captions vs simple prompts) changes FVD substantially (374 to 336).
Limitations
- Video quality depends heavily on the quality of the first-stage reference image and caption.
- Base model runs at low resolution (320x320) and 2 fps before interpolation and super-resolution.
Experimental setup
Datasets: WebVid-10M
Benchmarks: UCF-101 MSR-VTT
Metrics: FVD IS CLIPSIM
Baselines: Make-A-Video VideoFactory ModelScope Lavie VidRD PYoCo LVDM CogVideo MagicVideo Video LDM VideoComposer VideoFusion
Builds on: Stable Diffusion ControlNet Video LDM Make-A-Video SPADE
Closest papers by meaning
- TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.89)
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.89)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.88)
- LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity (CVPR 2025 · similarity 0.88)
- Grid Diffusion Models for Text-to-Video Generation (CVPR 2024 · similarity 0.87)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.