Research map / Video / Video generation
Mind the Time: Temporally-Controlled Multi-Event Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MinT, a DiT-based video generator fine-tuned on ~200k time-annotated videos that binds each event caption to a time span via a rescaled RoPE (ReRoPE) temporal cross-attention and scene-cut conditioning, enabling timed multi-event video generation.
Problem
Existing video generators take a single text prompt and tend to ignore events or misorder them when asked to generate multiple events; they also cannot control event timing.
Approach
Bind each event caption to a specific time range using a temporal cross-attention layer with rescaled RoPE positional encoding (all events rescaled to equal length), plus a learnable scene-cut token, so the model attends to one event at a time while keeping transitions smooth.
Key results
- On HoldOut T2V, MinT achieves higher event text alignment and transition smoothness than baselines while keeping visual quality comparable, with human preference favoring MinT on event-related metrics.
- On HoldOut I2V, MinT reaches FVD 218.21 versus 495.75 for MEVG and reduces cuts from 0.108 to 0.025 per video.
- In prompt enhancement on VBench, MinT attains dynamic degree 0.711 versus 0.481 for short prompts, with competitive quality metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MinT instead of MEVG | text-to-video multi-event generation · HoldOut | text-to-video alignment (TA): 2.72 -> 3.00 | Helped |
| use MinT instead of MEVG | text-to-video multi-event generation · StoryBench | text-to-video alignment (TA): 2.83 -> 3.37 | Helped |
| replace ReRoPE with concatenated time embeddings (ablation) | MinT · HoldOut | text-to-video alignment (TA): 2.92 -> 2.42 | Hurt |
| replace ReRoPE with hard attention masking (ablation) | MinT · HoldOut | temporal consistency (TC): 2.44 -> 2.30 | Hurt |
| replace ReRoPE with vanilla RoPE (ablation) | MinT · HoldOut | text-to-video alignment (TA): 2.92 -> 2.79 | Hurt |
| set ReRoPE rescale length to L=4 (ablation) | MinT · HoldOut | text-to-video alignment (TA): 2.92 -> 2.88 | No clear effect |
| set ReRoPE rescale length to L=16 (ablation) | MinT · HoldOut | text-to-video alignment (TA): 2.92 -> 2.90 | No clear effect |
| remove scene-cut conditioning (ablation) | MinT · HoldOut | temporal consistency (TC): 2.44 -> 2.34 | Hurt |
| remove scene-cut conditioning (ablation) | MinT · HoldOut | average cuts per video: 0.026 -> 0.084 | Hurt |
| add temporal captions to short prompts | MinT · VBench Dynamic Degree prompts | dynamic degree: 0.517 -> 0.711 | Helped |
Surprising findings
- Concatenating events into one prompt with CogVideoX and Mochi often generates only a subset of the events.
Limitations
- Evaluated on video with limited resolution (512x288) and up to 12 seconds
- Some failure cases discussed in appendix; relies on manually annotated timestamps for training data
Experimental setup
Datasets: video datasets from prior work [19, 100] annotated with temporal events StoryBench VBench prompt list
Benchmarks: HoldOut StoryBench VBench
Metrics: FID FVD CLIP-score VideoScore visual quality VideoScore dynamic degree VideoScore text alignment VideoScore temporal consistency number of cuts VBench subject consistency background consistency
Baselines: CogVideoX-5B Mochi 1 MEVG AutoReg Concat Kling 1.5 Gen-3 Alpha
Builds on: DiT RoPE rectified flow Positional Interpolation
Closest papers by meaning
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.86)
- VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis (ICLR 2025 · similarity 0.86)
- Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Grounding (CVPR 2025 · similarity 0.85)
- TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning (ICLR 2025 · similarity 0.85)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.85)
- Event-Equalized Dense Video Captioning (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.