Research map / Video / Video generation
Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Dysen-VDM uses ChatGPT-driven action planning, DSG conversion and scene imagination, then encodes the enriched dynamic scene graph with a recurrent graph Transformer to condition a latent video diffusion model for text-to-video generation.
Problem
Diffusion-based text-to-video models suffer from action occurrence disorder and crude motion because short prompts under-specify temporal dynamics of complex actions.
Approach
Use an LLM (ChatGPT) via in-context learning to plan time-ordered actions, convert them to a dynamic scene graph, and enrich it with imagined details; encode the DSG with a recurrent graph Transformer and inject it into the 3D-UNet of a latent video diffusion model.
Key results
- Zero-shot on UCF-101 Dysen-VDM reaches IS 35.57 and FVD 325.42, beating the listed baselines on both metrics.
- Fine-tuned on UCF-101, Dysen-VDM achieves IS 95.23 and FVD 255.42, versus 92.72 IS for Latent-Shift.
- Human evaluation on ActivityNet gives Dysen-VDM 92.4 scene richness and 86.6 action faithfulness, against 75.0 and 70.7 for the best competing systems.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Dysen-VDM instead of AlignLatent | UCF-101 zero-shot generation · UCF-101 | IS: 33.45 -> 35.57 | Helped |
| use Dysen-VDM instead of MakeVideo | UCF-101 zero-shot generation · UCF-101 | FVD: 367.23 -> 325.42 | Helped |
| use Dysen-VDM instead of MakeVideo | MSR-VTT zero-shot generation · MSR-VTT | FID: 13.17 -> 12.64 | Helped |
| use Dysen-VDM instead of MakeVideo | MSR-VTT zero-shot generation · MSR-VTT | CLIPSIM: 0.3049 -> 0.3204 | Helped |
| use Dysen-VDM instead of Latent-Shift | UCF-101 fine-tuned generation · UCF-101 | IS: 92.72 -> 95.23 | Helped |
| add the Dysen module (ablation) | Dysen-VDM · UCF-101 | FVD: 346.40 -> 255.42 | Helped |
| add the Dysen module (ablation) | Dysen-VDM · ActivityNet | FVD: 627.30 -> 485.48 | Helped |
| add scene imagination (ablation) | Dysen-VDM · UCF-101 | FVD: 332.92 -> 255.42 | Helped |
| add scene imagination (ablation) | Dysen-VDM · ActivityNet | FVD: 597.83 -> 485.48 | Helped |
| add sliding window context (SWC) (ablation) | Dysen-VDM · UCF-101 | FVD: 292.16 -> 255.42 | Helped |
| add sliding window context (SWC) (ablation) | Dysen-VDM · ActivityNet | FVD: 533.22 -> 485.48 | Helped |
| add RL optimization for in-context learning (ablation) | Dysen-VDM · UCF-101 | FVD: 319.01 -> 255.42 | Helped |
Surprising findings
- Removing the entire Dysen module raised FVD by 90.98 on UCF-101, the largest ablation drop, indicating motion-dynamics modeling matters most.
Limitations
- Relies on proprietary ChatGPT API for action planning and scene imagination, adding dependence on an external LLM.
- Code said to be released 'later'; no release link given in the paper.
Experimental setup
Datasets: WebVid UCF-101 MSR-VTT ActivityNet
Benchmarks: UCF-101 MSR-VTT ActivityNet
Metrics: Inception Score FVD FID CLIPSIM human evaluation TriRec
Baselines: CogVideo VDM Latent-VDM MagicVideo MakeVideo AlignLatent Latent-Shift VideoFactory InternVid
Builds on: Latent Video Diffusion Model dynamic scene graph in-context learning CLIP
Closest papers by meaning
- LLM-grounded Video Diffusion Models (ICLR 2024 · similarity 0.89)
- MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance (AAAI 2024 · similarity 0.88)
- ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation (ICLR 2025 · similarity 0.88)
- VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.88)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.87)
- VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.