Research map / Video / Video generation
Vlogger: Make Your Dream A Vlog
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Vlogger uses an LLM director (GPT-4) to plan a minute-level vlog into script, actors, shots and voice, and a new video diffusion model ShowMaker (SD-1.4-based, with spatial-temporal enhanced blocks and mixed generation/prediction training) to shoot each scene.
Problem
Generating minute-level vlogs with complex storylines and many scene changes is beyond short-clip text-to-video diffusion models, and existing long-video methods either need large long-video datasets or suffer from shot-change incoherence.
Approach
Decompose long video generation into LLM-driven top-down planning (script, actor design, protagonist assignment, duration scheduling) and bottom-up shooting, where each scene snippet is generated by ShowMaker conditioned on scene text and actor image, with autoregressive prediction to reach the allocated duration.
Key results
- Zero-shot UCF-101 FVD of 292.43 with hand-crafted prompts, better than Make-A-Video (367.23) and PYoCo (355.19).
- Zero-shot Kinetics-400 FID of 37.23 versus Phenaki's 37.74, using 66.7% of Phenaki's pre-training videos (10M vs 15M).
- Vlogger generates vlogs over 5 minutes (up to 7min59s in a visualization) with script and actor coherence; generation-plus-prediction ablation reports CLIP-I 0.6294 vs 0.5642 for autoregressive-only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Vlogger's planned scene-by-scene generation process instead of no autoregression (ablation) | Vlogger · Vlog generation | CLIP-I: 0.5683 -> 0.6294 | Helped |
| Use Vlogger's planned scene-by-scene generation process instead of only autoregressive generation (ablation) | Vlogger · Vlog generation | CLIP-I: 0.5642 -> 0.6294 | Helped |
| Use Vlogger's planned scene-by-scene generation process instead of only autoregressive generation (ablation) | Vlogger · Vlog generation | CLIP-T: 0.2535 -> 0.2756 | Helped |
| Add temporal text cross attention (ablation) | ShowMaker · Vimeo11k, 0/16 prediction | FVD: 348.36 -> 333.63 | Helped |
| Add temporal text cross attention (ablation) | ShowMaker · Vimeo11k, 1/15 prediction | FVD: 217.39 -> 178.27 | Helped |
| Add temporal text cross attention (ablation) | ShowMaker · Vimeo11k, 3/13 prediction | FVD: 297.37 -> 230.56 | Helped |
| Add temporal text cross attention (ablation) | ShowMaker · Vimeo11k, 5/11 prediction | FVD: 246.56 -> 193.52 | Helped |
| Add the mixed training paradigm to ShowMaker with TTCA (ablation) | ShowMaker · Vimeo11k, 0/16 prediction | FVD: 333.63 -> 257.70 | Helped |
| Add the mixed training paradigm to ShowMaker with TTCA (ablation) | ShowMaker · Vimeo11k, 1/15 prediction | FVD: 178.27 -> 123.51 | Helped |
| Add the mixed training paradigm to ShowMaker with TTCA (ablation) | ShowMaker · Vimeo11k, 3/13 prediction | FVD: 230.56 -> 118.02 | Helped |
| Add the mixed training paradigm to ShowMaker with TTCA (ablation) | ShowMaker · Vimeo11k, 5/11 prediction | FVD: 193.52 -> 109.31 | Helped |
| Add spatial image cross attention (ablation) | ShowMaker · COCO2017 | CLIP-I | Helped |
Surprising findings
- Phenaki-style fully autoregressive generation and no-autoregression baselines both lag Vlogger in CLIP-I even with the same scripts and videographer.
Limitations
- Relies on closed-source GPT-4 and multiple external foundation models in the pipeline.
- Evaluation of long vlogs is mostly qualitative; quantitative metrics are shown mainly for ablations on a self-collected benchmark.
Experimental setup
Datasets: LAION-400M WebVid-10M UCF-101 Kinetics-400 MSR-VTT Vimeo11k
Benchmarks: UCF-101 Kinetics-400 MSR-VTT Vimeo11k
Metrics: FVD FID CLIPSIM CLIP-FID CLIP-I CLIP-T vlog duration
Baselines: VideoFactory Make-A-Video PYoCo CogVideo Video LDM MagicVideo NUWA Phenaki TATS MovieFactory
Builds on: Stable Diffusion Make-A-Video Video LDM Phenaki MovieFactory
Closest papers by meaning
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.86)
- VlogReward: Learning Multi-Dimensional Evaluation for Vlog Editing (ICML 2026 · similarity 0.85)
- SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction (ICLR 2024 · similarity 0.83)
- VLog: Video-Language Models by Generative Retrieval of Narration Vocabulary (CVPR 2025 · similarity 0.83)
- VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis (ICLR 2025 · similarity 0.83)
- AgenticVBench: Can AI Agents Complete Real-World Video Production Tasks? (NeurIPS 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.