Research map / Video / Video generation
DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiTCtrl, a tuning-free method that uses mask-guided KV-sharing attention control and latent blending in CogVideoX (MM-DiT) to generate smooth multi-prompt longer videos, and introduces the MPVBench benchmark with a CSCV metric.
Problem
Multi-prompt longer video generation: existing text-to-video models produce isolated clips when given sequential prompts, while prior multi-prompt methods need heavy training data or are UNet-based and produce weak prompt following and unnatural transitions.
Approach
Analyzes MM-DiT 3D full attention and finds its text-video, video-text, and video-video regions behave like UNet cross/self/temporal attention, then uses mask-guided KV-sharing between consecutive prompt segments plus a latent blending strategy for transitions, with no training.
Key results
- DiTCtrl achieves the highest CSCV transition smoothness score of 84.90% on MPVBench, versus 84.37% for FreeNoise and 67.28% for Gen-L-Video.
- In a 28-user forced-ranking study, DiTCtrl scores 4.11 overall preference versus 3.81 for FreeNoise+DiT and 1.15 for Gen-L-Video.
- DiTCtrl reaches motion smoothness 97.80%, the highest among compared methods, while CLIP text-image similarity (30.68) trails FreeNoise (32.69) and Video-Infinity (32.35).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiTCtrl instead of FreeNoise | CogVideoX-2B multi-prompt video generation · MPVBench | CSCV: 84.37% -> 84.90% | Helped |
| Use DiTCtrl instead of FreeNoise+DiT | CogVideoX-2B multi-prompt video generation · MPVBench | Motion smoothness: 97.76% -> 97.80% | Helped |
| Use DiTCtrl instead of FreeNoise | CogVideoX-2B multi-prompt video generation · MPVBench | Text-Image similarity: 32.69% -> 30.68% | Hurt |
| Use DiTCtrl instead of FreeNoise+DiT | multi-prompt video generation · User study | Overall preference: 3.81 -> 4.11 | Helped |
| Add KV-sharing (ablation) | DiTCtrl · MPVBench | CSCV: 81.79% -> 84.90% | Helped |
| Add KV-sharing (ablation) | DiTCtrl · MPVBench | Motion smoothness: 97.35% -> 97.80% | Helped |
| Add KV-sharing (ablation) | DiTCtrl · MPVBench | Text-Image similarity: 31.37% -> 30.68% | Hurt |
| Add latent blending (ablation) | DiTCtrl · multi-prompt video generation | transition smoothness | Helped |
| Add KV-sharing (ablation) | DiTCtrl · multi-prompt video generation | motion coherence | Helped |
| Add mask guidance (ablation) | DiTCtrl · multi-prompt video generation | object attribute consistency | Helped |
Surprising findings
- DiTCtrl has lower CLIP text-image similarity than FreeNoise and Video-Infinity, which the authors attribute to KV-sharing carrying prior-segment content forward.
Limitations
- Open-source video models show weaker conceptual composition, causing attribute binding errors across segments.
- DiT-based architectures have high computational overhead, limiting inference speed.
- Text-image similarity is lower than some baselines due to KV-sharing trade-off.
Experimental setup
Datasets: DAVIS
Benchmarks: MPVBench
Metrics: CSCV CLIP text-image similarity VBench motion smoothness user preference score (forced ranking)
Baselines: Gen-L-Video FreeNoise Video-Infinity Kling FreeNoise+DiT
Builds on: MasaCtrl Prompt-to-Prompt MultiDiffusion FreeNoise CogVideoX
Closest papers by meaning
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.90)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.90)
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.90)
- ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models (CVPR 2025 · similarity 0.89)
- Mask^2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long Video Generation (CVPR 2025 · similarity 0.89)
- PEEKABOO: Interactive Video Generation via Masked-Diffusion (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.