Research map / Video / Video generation
Mask^2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mask2DiT adapts CogVideoX with a symmetric binary attention mask so each text prompt governs only its own video segment, plus a segment-level conditional mask for auto-regressive scene extension, for multi-scene long video generation.
Problem
Multi-scene long video generation with DiT-based text-to-video models, where concatenated multi-prompt scenes lose fine-grained alignment between each scene and its text, and scene count is fixed and hard to extend.
Approach
Concatenate text and video tokens of n scenes into one sequence, apply a symmetric binary attention mask so each text only attends to its own segment while all visual tokens attend to each other, and train a segment-level conditional mask that conditions the last segment on preceding ones for auto-regressive extension; a two-stage pretrain (concatenated Panda70M clips) then SFT (coherent multi-sho
Key results
- Mask2DiT reaches 70.95% Visual Consistency and FVD 720.01 versus 55.01% and 835.35 for CogVideoX, and 47.45% Sequence Consistency versus 38.82%.
- Mask2DiT attains 23.94% Semantic Consistency, second to StoryDiffusion's 25.59%.
- Ablation: the V2 attention mask improves Visual Consistency by 10.36% over the baseline mask V1 (68.45 vs 58.09).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Mask2DiT multi-scene training | CogVideoX · Multi-scene video evaluation set | Visual Cons. (%): 55.01 -> 70.95 | Helped |
| Add Mask2DiT multi-scene training | CogVideoX · Multi-scene video evaluation set | FVD: 835.35 -> 720.01 | Helped |
| Use the V2 attention mask instead of V1, allowing inter-video attention (ablation) | Mask2DiT · Multi-scene video evaluation set (batch size 1, 256×384) | Visual Cons. (%): 58.09 -> 68.45 | Helped |
| Use the V3 attention mask instead of V2 (ablation) | Mask2DiT · Multi-scene video evaluation set (batch size 1, 256×384) | Visual Cons. (%): 68.45 -> 77.56 | Helped |
| Use the V4 attention mask instead of V2 (ablation) | Mask2DiT · Multi-scene video evaluation set (batch size 1, 256×384) | Semantic Cons. (%): 23.82 -> 20.73 | Hurt |
| Use the V5 attention mask instead of V2 (ablation) | Mask2DiT · Multi-scene video evaluation set (batch size 1, 256×384) | Semantic Cons. (%): 23.82 -> 20.60 | Hurt |
| Increase the dataset filtering threshold from 0.0 to 0.8 (ablation) | Mask2DiT · Multi-scene video evaluation set (batch size 4, 256×384) | Visual Cons. (%): 63.94% to 73.22% | Helped |
| Increase the dataset filtering threshold from 0.0 to 0.8 (ablation) | Mask2DiT · Multi-scene video evaluation set (batch size 4, 256×384) | Semantic Cons. (%): 24.31% -> 23.54% | Hurt |
| Increase the dataset filtering threshold from 0.0 to 0.8 (ablation) | Mask2DiT · Multi-scene video evaluation set (batch size 4, 256×384) | Sequence Cons. (%): 44.13% -> 48.38% | Helped |
Surprising findings
- Richer inter-segment attention masks (V4, V5) raise Visual Consistency but reduce Semantic Consistency because each prompt attends to all segments.
- Stricter dataset filtering raises Visual and Sequence Consistency but lowers Semantic Consistency.
Limitations
- Fixed number of scenes (n=3) during the main training setup
- Evaluation relies on an in-house test set and ViClip-based metrics
- Semantic consistency lags StoryDiffusion
Experimental setup
Datasets: Panda70M long-form videos (PySceneDetect shots, ViClip-filtered, Gemini-annotated) ChatGPT-generated evaluation prompts
Benchmarks: in-house multi-scene evaluation set (50 scenes, 3 prompts each)
Metrics: Visual Consistency (ViClip) Semantic Consistency Sequence Consistency FVD
Baselines: CogVideoX StoryDiffusion TALC VideoStudio
Builds on: CogVideoX DiT v-prediction
Closest papers by meaning
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.89)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.88)
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model (NeurIPS 2024 · similarity 0.88)
- Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation (CVPR 2025 · similarity 0.88)
- MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers (CVPR 2024 · similarity 0.88)
- VDT: General-purpose Video Diffusion Transformers via Mask Modeling (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.