Research map / 3D vision / 3D generation
ChatCam: Empowering Camera Control through Conversational AI
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ChatCam, an LLM-agent system that turns natural-language camera instructions into 3D camera trajectories via a GPT-based autoregressive generator (CineGPT) and a CLIP-based anchor determinator, rendered on 3DGS.
Problem
Camera trajectory generation for 3D scenes from human language guidance has not been explored; existing video-generation camera control needs user-provided trajectories and 3D grounding methods are costly.
Approach
Quantize camera trajectories into discrete tokens with a VQ-VAE tokenizer and train a GPT-style decoder on a text-trajectory pairs so text can directly generate trajectories; anchor the trajectories to scene objects via CLIP-based image retrieval and gradient-based refinement; an LLM agent (GPT-4) plans and composes atomic trajectories.
Key results
- Full ChatCam with GPT-4 and anchor determination achieves translation MSE 5.3 and rotation MSE 2.9, versus LERF (17.7 / 4.9) and SA3D (19.5 / 6.3).
- User study: full model scores 84.9 visual quality and 67.9 alignment, versus LERF's 9.4 and 28.3 and SA3D's 5.7 and 3.8.
- Removing anchor determination raises GPT-4 translation MSE from 5.3 to 16.2 and rotation MSE from 2.9 to 8.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ChatCam instead of LERF | Camera trajectory evaluation · Camera trajectory evaluation | Translation MSE: 17.7 -> 5.3 | Helped |
| Use ChatCam instead of LERF | Camera trajectory evaluation · Camera trajectory evaluation | Rotation MSE: 4.9 -> 2.9 | Helped |
| Use ChatCam instead of LERF | Camera trajectory evaluation · Camera trajectory evaluation | Visual Quality: 9.4 -> 84.9 | Helped |
| Use ChatCam instead of LERF | Camera trajectory evaluation · Camera trajectory evaluation | Alignment: 28.3 -> 67.9 | Helped |
| Add anchor determination (ablation) | ChatCam · Camera trajectory evaluation | Translation MSE: 16.2 -> 5.3 | Helped |
| Add anchor determination (ablation) | ChatCam · Camera trajectory evaluation | Rotation MSE: 8.5 -> 2.9 | Helped |
| Use GPT-4 instead of LLaMA-2 as the LLM agent (ablation) | ChatCam · Camera trajectory evaluation | Translation MSE: 6.4 -> 5.3 | Helped |
| Use GPT-4 instead of LLaMA-2 as the LLM agent (ablation) | ChatCam · Camera trajectory evaluation | Rotation MSE: 3.6 -> 2.9 | Helped |
| Use GPT-4 instead of GPT-3.5 as the LLM agent (ablation) | ChatCam · Camera trajectory evaluation | Translation MSE: 7.3 -> 5.3 | Helped |
| Use GPT-4 instead of GPT-3.5 as the LLM agent (ablation) | ChatCam · Camera trajectory evaluation | Rotation MSE: 3.5 -> 2.9 | Helped |
Surprising findings
- With anchor determination, the LLaMA-2 agent (translation MSE 6.4) and GPT-3.5 agent (7.3) come close to GPT-4 on trajectory accuracy, and GPT-3.5 gets a lower rotation MSE (3.5) than the full GPT-4 model (2.9 is lower, so GPT-4 still best there).
Limitations
- No established direct baseline; comparisons use adapted LERF/SA3D agents.
- Text-trajectory dataset is small (about 1000 manually built Blender trajectories).
Experimental setup
Datasets: mip-NeRF 360 OMMO Hypersim MannequinChallenge Blender-constructed text-trajectory dataset (~1000 trajectories)
Benchmarks: mip-NeRF 360 OMMO Hypersim MannequinChallenge
Metrics: translation MSE rotation MSE visual quality (user study) alignment (user study)
Baselines: LERF SA3D
Builds on: VQ-VAE GPT CLIP 3D Gaussian Splatting LERF
Closest papers by meaning
- Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation (NeurIPS 2026 · similarity 0.86)
- HSI-GPT: A General-Purpose Large Scene-Motion-Language Model for Human Scene Interaction (CVPR 2025 · similarity 0.85)
- CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation (NeurIPS 2025 · similarity 0.85)
- Unveiling the Value of Motion for Cinematic Camera Trajectories (NeurIPS 2026 · similarity 0.84)
- ChatHuman: Chatting about 3D Humans with Tools (CVPR 2025 · similarity 0.84)
- PreciseCam: Precise Camera Control for Text-to-Image Generation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.