Research map / Video / Video generation
LLM-grounded Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LVD, a training-free pipeline where an LLM (GPT-4/GPT-3.5) generates spatiotemporal bounding-box layouts from a text prompt, which then guide ModelScope video diffusion via attention-based energy guidance.
Problem
Text-to-video diffusion models struggle to follow intricate spatiotemporal prompts, producing wrong object counts, attributes, spatial layouts, and motion.
Approach
Use a text-only LLM to produce dynamic scene layouts (per-frame boxes with IDs) from the prompt with few in-context examples, then steer a frozen video diffusion model's cross-attention maps into the boxes using a top-k energy plus a center-of-mass energy applied as classifier-style guidance at each denoising step.
Key results
- GPT-4 generated DSLs reach about 98% accuracy on the benchmark, versus about 77% for GPT-3.5 and low accuracy for the retrieval baseline.
- LVD significantly outperforms the ModelScope base model on the spatiotemporal benchmark by a large margin in rule-based text-video alignment.
- Human evaluators preferred or rated LVD on par with the baseline in 96.8% of alignment cases and 85.5% of visual quality cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4 instead of GPT-3.5 for DSL generation | LLM-grounded DSL generation · Proposed DSL benchmark | average accuracy: 77% -> 98% | Helped |
| Add LLM-generated dynamic scene layouts and DSL guidance | ModelScope · Proposed video benchmark | text-video alignment | Helped |
| Add LLM-generated dynamic scene layouts and DSL guidance | ModelScope · UCF-101 and MSR-VTT | FVD | Helped |
| Compare LVD with the baseline in human evaluation | ModelScope · Evaluator-based assessment | alignment cases where LVD outperforms or is at least on par: 96.8% | Helped |
| Compare LVD with the baseline in human evaluation | ModelScope · Evaluator-based assessment | visual quality cases where LVD outperforms or is at least on par: 85.5% | Helped |
| Add the Center-of-Mass energy term to E_topk (ablation) | LVD DSL-grounded video generation · Sequential actions | text-video alignment | Helped |
| Use only E_topk instead of the full energy function (ablation) | LVD DSL-grounded video generation · Proposed video benchmark | text-video alignment | Hurt |
| Replace E_topk with the ratio-based energy E_ratio (ablation) | LVD DSL-grounded video generation · Proposed video benchmark at 512×512 | text-video alignment | Hurt |
| Add E_corner to E_topk (ablation) | LVD DSL-grounded video generation · Proposed video benchmark at 512×512 | text-video alignment | No clear effect |
| Increase E_corner weight from 1× to 10× (ablation) | LVD DSL-grounded video generation · Proposed video benchmark at 512×512 | text-video alignment: 59.2% and 60.9% | No clear effect |
| Remove spatial normalization from E_corner (ablation) | LVD DSL-grounded video generation · Proposed video benchmark at 512×512 | text-video alignment | Hurt |
| Use one in-context example instead of three (ablation) | GPT-4 DSL generation · Proposed DSL benchmark | DSL quality | Hurt |
Surprising findings
- Text-only LLMs given only three in-context examples (one per property) generalize to unseen physics such as elasticity differences (rocks not bouncing) and air friction without explicit cues.
- Interestingly, GPT-3.5 layouts yielded better text-video alignment on three tasks because they involve less motion, making videos easier to generate.
Limitations
- Inherits base model limitations such as poor artistic styles and object synthesis quality (e.g., apple cartoon failure).
- Attention-based control can cause ambiguous energy minimization (e.g., hiding an object rather than moving it).
- LLM queries are independent, so long-horizon sequential layouts occasionally fail.
Experimental setup
Datasets: WebVid-2M UCF-101 MSR-VTT
Benchmarks: LVD spatiotemporal prompt benchmark (generative numeracy, attribute binding, visibility, spatial dynamics, sequential ac UCF-101 MSR-VTT
Metrics: success rate (rule-based, OWL-ViT detection) FVD human preference rate
Baselines: ModelScope retrieval-based layout baseline (Frozen-In-Time + GLIP + SORT) Boxdiff Text2Video-Zero Make-A-Video
Builds on: ModelScope classifier guidance Boxdiff LMD
Closest papers by meaning
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.91)
- MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance (AAAI 2024 · similarity 0.90)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.90)
- Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs (CVPR 2024 · similarity 0.89)
- Grid Diffusion Models for Text-to-Video Generation (CVPR 2024 · similarity 0.89)
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.