Research map / Video / Video generation
Compositional 3D-aware Video Generation with LLM Director
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C3V, which uses an LLM as director to decompose a text prompt into scene, object, and motion concepts, builds each as 3D Gaussians, then composes them via multimodal-LLM trajectory guidance and SDS refinement with 2D diffusion priors to produce 3D-aware videos.
Problem
Precisely controlling individual concepts (character motion and appearance, viewpoint movement) in text-to-video generation, which implicit 2D methods handle poorly, and composing multiple concepts plausibly.
Approach
Treat the LLM as a director that assigns expert models to produce separate 3D Gaussian representations per concept, then compose them in 3D using step-by-step multimodal-LLM trajectory estimation lifted to 3D and refined with Score Distillation Sampling from a 2D diffusion model.
Key results
- C3V achieves the best QAlign video quality of 4.112 and CLIP score of 38.36, versus 3.868 and 35.20 for VideoCrafter2.
- Baselines 4D-FY and Comp4D score far lower (QAlign video quality 2.154 and 2.142) on the paper's complex multi-character prompts.
- Ablations show step-by-step trajectory estimation plus bounding-box representation yields smoother trajectories than direct GPT-4V prompting, with SDS refinement improving scale, location and orientation (qualitative only).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use C3V instead of VideoCrafter2 | Generated videos · Generated videos | QAlign-img-quality: 3.839 -> 4.030 | Helped |
| Use C3V instead of VideoCrafter2 | Generated videos · Generated videos | QAlign-img-aesthetic: 3.199 -> 3.471 | Helped |
| Use C3V instead of VideoCrafter2 | Generated videos · Generated videos | QAlign-vid-quality: 3.868 -> 4.112 | Helped |
| Use C3V instead of VideoCrafter2 | Generated videos · Generated videos | QAlign-vid-aesthetic: 3.159 -> 3.723 | Helped |
| Use C3V instead of VideoCrafter2 | Generated videos · Generated videos | CLIP Score: 35.20 -> 38.36 | Helped |
| Represent objects with bounding boxes rather than relying on a direct trajectory prompt (ablation) | GPT-4V trajectory estimation · Object trajectory estimation | trajectory quality | Helped |
| Estimate start and end points, then reason the path step by step, rather than prompting directly for a trajectory (ablation) | GPT-4V trajectory estimation · Object trajectory estimation | trajectory quality | Mixed |
| Combine bounding-box guidance with step-by-step trajectory estimation (ablation) | GPT-4V trajectory estimation · Object trajectory estimation | trajectory quality | Helped |
| Add SDS scale refinement (ablation) | C3V composition · Alien walking in front of cabin door | composition quality | Helped |
| Add SDS location refinement (ablation) | C3V composition · Alien walking in front of cabin door | composition quality | Helped |
| Add rotation refinement so the object faces its direction of travel (ablation) | C3V composition · Alien walking in front of cabin door | orientation quality | Helped |
Limitations
- Instantiated with only three concepts (scene, humanoid object, human motion)
- Composition relies on rough LLM trajectory priors and 2D diffusion SDS rather than physical dynamics
- Long multi-scene video generation not yet explored
Experimental setup
Datasets: Motion-X
Metrics: Q-Align image quality Q-Align image aesthetic Q-Align video quality Q-Align video aesthetic CLIP score
Baselines: 4D-FY Comp4D VideoCrafter2
Builds on: DreamFusion 3D Gaussian Splatting Score Distillation Sampling LucidDreamer HumanGaussian
Closest papers by meaning
- VideoTetris: Towards Compositional Text-to-Video Generation (NeurIPS 2024 · similarity 0.90)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.89)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.89)
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model (NeurIPS 2024 · similarity 0.89)
- LLM-grounded Video Diffusion Models (ICLR 2024 · similarity 0.88)
- EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.