Research map / Video / Video generation
Vivid-ZOO: Multi-View Video Generation with Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Vivid-ZOO factors text-to-multi-view-video generation into a viewpoint-space (MVDream) and a time (AnimateDiff) component, training only lightweight 3D-2D and 2D-3D alignment layers on a new 14,271-video captioned multi-view dataset (MV-VideoNet).
Problem
Text-to-multi-view-video (T2MVid) generation is underexplored, due to the lack of large captioned multi-view video datasets and the complexity of modeling four-dimensional (view, time, 2D space) data.
Approach
Reuse and freeze the temporal layers of AnimateDiff and the spatial/multi-view layers of MVDream, bridging their domain gap with zero-initialized MLP 3D-2D and 2D-3D alignment layers, so only the alignment layers are trained.
Key results
- Ours achieves FVD 1311.12 vs 2038.66 for MVDream + IP-AnimateDiff, with 72% human preference over the baseline.
- MVDream + IP-AnimateDiff has slightly higher CLIP score (32.71 vs 32.14).
- Training only alignment layers reduces training cost to about 2 days on 8 A100 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MVDream + IP-AnimateDiff with Vivid-ZOO | Multi-view video generation · Multi-view video generation | FVD: 2038.66 -> 1311.12 | Helped |
| Replace MVDream + IP-AnimateDiff with Vivid-ZOO | Multi-view video generation · Multi-view video generation | CLIP: 32.71 -> 32.14 | Hurt |
| Replace MVDream + IP-AnimateDiff with Vivid-ZOO | Multi-view video generation · Multi-view video generation | Overall: 28% -> 72% | Helped |
| Add the pretrained multi-view image diffusion model as the spatial module instead of Stable Diffusion 1.5 (ablation) | Vivid-ZOO · Multi-view video generation | Overall: 44.25% -> 81.00% | Helped |
| Use the multi-view temporal module with 3D-2D and 2D-3D alignments instead of TM LoRA (ablation) | Vivid-ZOO · Multi-view video generation | Overall: 10.00% -> 81.00% | Helped |
| Add 3D-2D alignment (ablation) | Vivid-ZOO · Multi-view video generation | Overall: 53.50% -> 81.00% | Helped |
| Add 2D-3D alignment (ablation) | Vivid-ZOO · Multi-view video generation | Overall: 55.50% -> 81.00% | Helped |
Surprising findings
- Naively combining pre-trained multi-view image and 2D temporal layers yields poor results due to domain gap between synthetic multi-view and real 2D video data.
- Adding LoRA to 2D temporal attention (TM LoRA) produced low-quality results, with only 10% human preference in ablation.
Limitations
- Quantitative evaluation of multi-view consistency is an open problem; relies on FVD, CLIP and user study.
- Dataset is rendered from synthetic animated Objaverse objects and manually filtered.
- Generated resolution is 256x256 during training.
Experimental setup
Datasets: Objaverse 4D animated objects MV-VideoNet
Benchmarks: MV-VideoNet
Metrics: FVD CLIP score human preference (overall)
Baselines: MVDream + IP-AnimateDiff w/o MS w SD w/o MT w TM LoRA w/o 3D-2D alignment w/o 2D-3D alignment
Builds on: MVDream AnimateDiff Stable Diffusion ControlNet Cap3D
Closest papers by meaning
- Grid Diffusion Models for Text-to-Video Generation (CVPR 2024 · similarity 0.91)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.91)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.90)
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.90)
- VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide (CVPR 2025 · similarity 0.90)
- 4Diffusion: Multi-view Video Diffusion Model for 4D Generation (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.