Research map / Video / Video generation
Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Auteur maps natural-language shot descriptions to actor-relative camera framing via a fine-tuned Qwen2.5-VL director that emits a discrete DSL of sparse keyframes, deterministically decoded into 6-DoF camera trajectories that condition video generators.
Problem
Generative video models offer little intentional, human-aware camera control; world-space 6-DoF trajectories are decoupled from the actor, so camera motion often fails to frame the subject.
Approach
Parameterize camera state relative to the actor's body frame (camera level, look-at level, shot scale, framing offset, orientation angle, Dutch angle), discretize it into a cinematographic DSL, and have an LLM generate actor and camera DSL programs whose sparse keyframes are interpolated into dense trajectories.
Key results
- On the controlled benchmark, Auteur reaches Auteur Score 0.901 and Out-Rate 5.45% versus 0.661 and 36.2% for PulpMotion.
- On PulpMotion pure split, Auteur achieves CLaTr 71.1 and out-rate 1.26% versus 7.00% for LAMP.
- In a blind user study with 24 participants, Auteur won 82.7% (quality) and 80.4% (adherence) against LAMP.
Surprising findings
- LAMP attains the highest Cam F1 (99.7) on the PulpMotion pure split despite a higher out-of-frame rate than Auteur.
- A world-space DSL fitted to ground-truth trajectories reaches only 0.280 CamF1 versus 0.623 for the actor-relative representation.
Limitations
- DSL keyframe vocabulary may under-represent continuous or reactive shot styles such as handheld or continuous reframing.
- Actor modeled only as coarse trajectory without fine articulation.
- Final perceptual quality largely determined by the downstream video generator.
Experimental setup
Datasets: CondensedMovies SOMA motion library (procedural synthesis) PulpMotion benchmark
Benchmarks: Auteur controlled diagnostic benchmark PulpMotion (pure and mixed) VBench
Metrics: F-Ori F-RoT F-Scale F-Tilt F-Roll Auteur Score CamF1 Out-Rate CLaTr FD_frame
Baselines: LAMP PulpMotion World-space DSL (GT recon.) Wan2.2 TI2V 5B
Builds on: Qwen2.5-VL TRAM SOMA Plücker embedding VACE
Closest papers by meaning
- Unveiling the Value of Motion for Cinematic Camera Trajectories (NeurIPS 2026 · similarity 0.89)
- CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation (NeurIPS 2025 · similarity 0.89)
- ONE-SHOT: Compositional Human-Environment Video Synthesis via Spatial-Decoupled Motion Injection and Hybrid Context Integration (NeurIPS 2026 · similarity 0.88)
- HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation (NeurIPS 2024 · similarity 0.88)
- TokenMotion: Decoupled Motion Control via Token Disentanglement for Human-centric Video Generation (CVPR 2025 · similarity 0.88)
- CineOrchestra: Unified Entity-Centric Conditioning for Cinematic Video Generation (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.