Research map / Video / Video generation
VideoBooth: Diffusion-based Video Generation with Image Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoBooth, a feed-forward text-and-image-prompt video generator that combines CLIP-based coarse embeddings with attention-injected multi-scale fine image features into a pretrained video diffusion model, and introduces a 48.7K-pair WebVid-derived dataset.
Problem
Text prompts alone cannot precisely specify a subject's appearance for customized video generation; prior image-prompt methods target images or require finetuning at inference.
Approach
Inject image prompts at two levels: coarse CLIP image embeddings mapped into text-embedding space via an MLP, and fine multi-scale VAE latents of the image prompt appended as extra keys/values in cross-frame attention so the first frame's details propagate to later frames; train coarse then fine.
Key results
- VideoBooth achieves CLIP-Image 74.80 and DINO 65.10 vs ELITE 73.75 and 58.95 on the 650-pair test set, with CLIP-Text comparable (30.10).
- User study with 25 participants prefers VideoBooth on image alignment, text alignment, and overall quality over Textual Inversion, DreamBooth, and ELITE.
- Ablation: full model reaches CLIP-Image 76.16 and DINO 69.74, beating coarse-only, fine-only, and unified-training variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VideoBooth instead of ELITE | VideoBooth · VideoBooth test benchmark | CLIP-Image: 73.7518 -> 74.7971 | Helped |
| use VideoBooth instead of ELITE | VideoBooth · VideoBooth test benchmark | DINO: 58.9522 -> 65.0979 | Helped |
| use VideoBooth instead of DreamBooth | VideoBooth · VideoBooth test benchmark | CLIP-Text: 30.6877 -> 30.0967 | No clear effect |
| add fine visual embeddings via Attention Injection to coarse embeddings (ablation) | VideoBooth · VideoBooth ablation subset | CLIP-Image: 75.4366 -> 76.1631 | Helped |
| add fine visual embeddings via Attention Injection to coarse embeddings (ablation) | VideoBooth · VideoBooth ablation subset | DINO: 64.9568 -> 69.7374 | Helped |
| add coarse visual embeddings via Image Encoder to fine embeddings (ablation) | VideoBooth · VideoBooth ablation subset | CLIP-Image: 75.5553 -> 76.1631 | Helped |
| add coarse visual embeddings via Image Encoder to fine embeddings (ablation) | VideoBooth · VideoBooth ablation subset | DINO: 66.0378 -> 69.7374 | Helped |
| train Image Encoder and Attention Injection together instead of coarse-to-fine (ablation) | VideoBooth · VideoBooth ablation subset | CLIP-Image: 76.1631 -> 75.8254 | Hurt |
| train Image Encoder and Attention Injection together instead of coarse-to-fine (ablation) | VideoBooth · VideoBooth ablation subset | DINO: 69.7374 -> 67.4201 | Hurt |
Surprising findings
- DreamBooth gets much higher CLIP-Text score but the authors attribute this to inserting the token rather than replacing it, which weakens image alignment.
Limitations
- Trained on image prompts with views similar to target videos, so back-view prompts cannot yield front-view videos.
- Dataset is limited to 48,724 pairs and a subset of categories (animals and cars) from the first 2.5M WebVid videos.
- Evaluation relies on automatic CLIP/DINO metrics and a 25-person user study on a small benchmark.
Experimental setup
Datasets: WebVid WebVid-10M
Benchmarks: VideoBooth test benchmark (650 pairs)
Metrics: CLIP-Text CLIP-Image DINO similarity user preference
Baselines: Textual Inversion DreamBooth ELITE
Builds on: Stable Diffusion Tune-A-Video ELITE CLIP Grounded-SAM
Closest papers by meaning
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.88)
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.87)
- VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models (NeurIPS 2024 · similarity 0.87)
- BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations (CVPR 2025 · similarity 0.87)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.87)
- Optical-Flow Guided Prompt Optimization for Coherent Video Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.