Research map / Video / Video generation
MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MovieDreamer, a hierarchical framework where a LLaMA-7B autoregressive model predicts compressed visual tokens from multimodal scripts, which a finetuned SDXL diffusion decoder renders into keyframes and an image-to-video model expands into long videos.
Problem
Video generation with diffusion models handles only short clips and struggles with complex narratives and long-term character consistency needed for long-form movie production.
Approach
Combine an autoregressive LLM that predicts a few continuous visual tokens per keyframe conditioned on a multimodal script (text descriptions, face embeddings, history) with ID-preserving diffusion decoding and anchor-based image-to-video extension for hour-long coherent generation.
Key results
- On the held-out test set, Ours-ref reaches FID 1.912 and LT consistency 0.893 versus StoryDiffusion's FID 4.643 and LT 0.596.
- On full-length video generation, Ours achieves CLIP 19.520 and Aesthetic 6.049, above StreamingT2V, SEINE, and FreeNoise on those metrics.
- Removing continuous token supervision (noC) lowers ST from 0.646 to 0.606 and LT from 0.814 to 0.755.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add reference-frame conditioning (ablation) | MovieDreamer · Story generation test set | LT consistency: 0.814 -> 0.893 | Helped |
| Remove continuous token supervision (ablation) | MovieDreamer · Story generation test set | LT consistency: 0.814 -> 0.755 | Hurt |
| Use 8 rather than 2 compressed tokens (ablation) | MovieDreamer image reconstruction · Image reconstruction | CLIP-sim: 0.825 -> 0.841 | Helped |
| Use 8 rather than 2 compressed tokens (ablation) | MovieDreamer image reconstruction · Image reconstruction | Inception Score: 8.235 -> 8.193 | No clear effect |
| Add L1 and L2 losses to continuous token supervision (ablation) | MovieDreamer autoregressive model · Story generation test set | LT consistency: 0.701 -> 0.814 | Helped |
| Add continuous token supervision to L1 and L2 training (ablation) | MovieDreamer autoregressive model · Story generation test set | LT consistency: 0.755 -> 0.814 | Helped |
| Use MovieDreamer rather than StoryDiffusion | Story generation · Story generation test set | LT consistency: 0.596 -> 0.814 | Helped |
| Use MovieDreamer rather than SEINE | Full-length video generation · Full-length video generation | CLIP-sim: 0.688 -> 0.704 | Helped |
Surprising findings
- Ablation on compressed tokens shows 8 tokens only slightly improve reconstruction over 2 tokens (CLIP-sim 0.841 vs 0.825), so the authors chose 2 tokens for sequence length.
- Using only L1/L2 losses or only continuous-token NLL degrades ID consistency or places characters at image edges, indicating the loss combination matters.
Limitations
- Confuses character IDs when many characters appear in one frame.
- Compressing each description sentence into one LongCLIP token loses information and object/scene consistency is limited.
- Requires substantial data and compute for training.
Experimental setup
Datasets: Condensed Movies OpenImages JourneyDB Object365
Benchmarks: held-out 100-movie test set (1M keyframes)
Metrics: CLIP score Inception Score Aesthetic Score FID ST consistency LT consistency CLIP-sim
Baselines: StoryDiffusion StoryGen StreamingT2V SEINE FreeNoise
Builds on: LLaMA SDXL Stable Video Diffusion LongCLIP LLaVA
Closest papers by meaning
- SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction (ICLR 2024 · similarity 0.89)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.88)
- StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation (NeurIPS 2024 · similarity 0.88)
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model (NeurIPS 2024 · similarity 0.88)
- MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation (CVPR 2025 · similarity 0.88)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.