Research map / Video / Video generation
CineOrchestra: Unified Entity-Centric Conditioning for Cinematic Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CineOrchestra, a single video DiT that jointly controls subjects, events, camera, and shot transitions via entity-centric conditioning and two parameter-free coordinated RoPEs, evaluated on new CineBench and CineBenchSyn benchmarks.
Problem
Cinematic video needs joint fine-grained control over multiple subjects, time-localized events, camera motion, and shot transitions, but existing text-to-video methods address these axes only in isolation.
Approach
Represent every cinematic element (subjects, camera, transitions) as an entity acting over a time interval within one entity-centric conditioning primitive, and handle it with an interval-sampled temporal RoPE (duration-normalized via beta(L)) plus a 2D entity-temporal cross-attention RoPE, adding no parameters to the base video DiT.
Key results
- On CineBench, CineOrchestra reaches transition timing recall 0.486 vs 0.431 for the next best (Phantom) and dense caption following ViCLIP 0.235 subject / 0.193 camera / 0.150 transition, the best in those columns.
- Subject identity M-DINO 0.502 is second only to Phantom (0.509), while global caption M-CLIP ties for second at 0.295.
- Ablation: removing the beta(L) similarity-peak rescaling drops M-DINO from 0.502 to 0.477 and transition recall from 0.486 to 0.412.
Surprising findings
- Phantom and VACE, which are single-shot personalization models, remain strongest on subject identity and global caption metrics, so the joint model does not dominate every metric.
- Partial 2D entity-temporal RoPE variants are not uniformly better than the baseline on every metric, suggesting the full coordinate design is needed.
Limitations
- Camera and transition control is coarse, relying on natural-language descriptors rather than explicit geometric pose.
- Clip length is bounded by the DiT context length (10 s training window), so long-form generation is not handled.
- No audio modality.
Experimental setup
Datasets: licensed movie and TV chunks (one-minute) generic video-caption pairs (pretraining) CineBench CineBenchSyn
Benchmarks: CineBench CineBenchSyn
Metrics: M-DINO M-CLIP ViCLIP VLM transition recall pairwise human preference win rate
Baselines: Phantom VACE CineTrans EchoShot MultiShotMaster ShotStream
Builds on: Video Alchemist Qwen-Image WeRoPE rectified flow Wan 2.1
Closest papers by meaning
- Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation (NeurIPS 2026 · similarity 0.88)
- EntityBench: Towards Entity-Consistent Long-Range Multi-Shot Video Generation (NeurIPS 2026 · similarity 0.88)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.87)
- CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation (NeurIPS 2025 · similarity 0.87)
- Stable Cinemetrics : Structured Taxonomy and Evaluation for Professional Video Generation (NeurIPS 2025 · similarity 0.87)
- Unveiling the Value of Motion for Cinematic Camera Trajectories (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.