Research map / Video / Video generation
Modular-Cam: Modular Dynamic Camera-view Video Generation with LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Modular-Cam uses an LLM director to split complex prompts into scenes with camera transitions, then generates each scene with an AnimateDiff-style video generator plus modular LoRA CamOperators for camera motion and an AdaControlNet for cross-scene consistency.
Problem
Text-to-video generation from complex prompts with multiple scenes and camera-view transformations; existing models produce near-static videos, entangled scenes, abrupt transitions, and poor camera-movement control.
Approach
Decompose a multi-scene instruction with an LLM into scenes and camera actions; control each scene's camera motion with a separately trained LoRA "CamOperator" selected from a motion pool, and chain scenes autoregressively with an AdaControlNet conditioned on the previous scene's last frame, using adaptive per-channel color normalization and randomized blending.
Key results
- Modular-Cam reports Dynamic Degree 0.994 versus 0.907 for the next best (StreamingT2V) and CLIP score 0.232 versus 0.224.
- Modular-Cam achieves the best User Rank of 1.7 (lower is better) among compared methods, with 15 participants.
- Imaging Quality of Modular-Cam is 0.546, below AnimateDiff (0.622) and FreeNoise (0.618).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Modular-Cam instead of FreeNoise | Multi-scene video generation · Self-generated multi-scene instructions | Motion Smoothness: 0.986 -> 0.988 | Helped |
| use Modular-Cam instead of StreamingT2V | Multi-scene video generation · Self-generated multi-scene instructions | Dynamic Degree: 0.907 -> 0.994 | Helped |
| use Modular-Cam instead of AnimateDiff | Multi-scene video generation · Self-generated multi-scene instructions | Imaging Quality: 0.622 -> 0.546 | Hurt |
| use Modular-Cam instead of StreamingT2V | Multi-scene video generation · Self-generated multi-scene instructions | CLIP: 0.224 -> 0.232 | Helped |
| use Modular-Cam instead of StreamingT2V | Multi-scene video generation · Self-generated multi-scene instructions | User Rank: 2.3 -> 1.7 | Helped |
| add LLM decomposition of the original instruction (ablation) | Modular-Cam · Self-generated multi-scene instructions | Motion Smoothness: 0.950 -> 0.988 | Helped |
| add LLM decomposition of the original instruction (ablation) | Modular-Cam · Self-generated multi-scene instructions | Dynamic Degree: 0.771 -> 0.994 | Helped |
| add LLM decomposition of the original instruction (ablation) | Modular-Cam · Self-generated multi-scene instructions | Imaging Quality: 0.511 -> 0.546 | Helped |
| add LLM decomposition of the original instruction (ablation) | Modular-Cam · Self-generated multi-scene instructions | CLIP: 0.221 -> 0.232 | Helped |
| add randomized blending (ablation) | AdaControlNet · Multi-scene video transitions | Color consistency | Helped |
| add adaptive pixel normalization (ablation) | AdaControlNet · Multi-scene video transitions | Color consistency | Helped |
| add LLM decomposition of the multi-scene instruction (ablation) | Modular-Cam · Two-scene field-and-house instruction | Scene adherence | Helped |
Surprising findings
- Imaging Quality is lower for Modular-Cam than AnimateDiff and FreeNoise, which the authors attribute to a content-richness versus quality trade-off.
Limitations
- Camera control is limited to six coarse motion patterns; fine-grained amounts of motion or rotation are not supported.
- Imaging quality trails some baselines, attributed to multi-scene complexity.
- Evaluation relies on a self-generated 1000-instruction set and a small user study (15 participants, 20 sets).
Experimental setup
Datasets: WebVid-10M
Benchmarks: self-generated 1000 multi-scene instruction set
Metrics: Motion Smoothness Dynamic Degree Imaging Quality CLIP score User Rank
Baselines: AnimateDiff FreeNoise SparseCtrl StreamingT2V
Builds on: AnimateDiff Stable Diffusion ControlNet LoRA
Closest papers by meaning
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.88)
- Training-free Camera Control for Video Generation (ICLR 2025 · similarity 0.88)
- Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control (NeurIPS 2024 · similarity 0.87)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.87)
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model (NeurIPS 2024 · similarity 0.87)
- ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.