Research map / 3D vision / Human pose estimation
Cinematic Behavior Transfer via NeRF-based Differentiable Filming
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Estimates world-coordinate camera trajectories and SMPL human tracks from a film shot by using a dynamic NeRF as a differentiable renderer, then transfers the motion to new 2D or 3D content via a 3D engine workflow.
Problem
Recovering accurate camera trajectories and 3D character motion in world coordinates from dynamic film shots, where SLAM and SMPL-based pipelines produce misaligned or implausible results, so that cinematic behavior can be transferred to new content.
Approach
Treats a trained dynamic NeRF of the SMPL tracks as a differentiable renderer to optimize a continuous SE(3) camera trajectory using composition and joint losses, then re-estimates SMPL tracks with the refined camera and transfers them through a 3D engine pipeline.
Key results
- On the frame composition metrics, the method reaches PA 94.8, IoU 94.0, and MPJPE 23.8 on push-in/pull-out shots, outperforming DROID-SLAM and iNeRF across shot types.
- In a user study on 30 shots with 10 volunteers, 2D camera movement rated 6.0±0.5 versus 4.7±1.3 for SLAM+SMPL on a seven-point Likert scale.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize camera trajectories with NeRF-based differentiable rendering and composition/joint losses instead of using the strongest baseline | Cinematic behavior transfer · PUSH-IN | PA: 89.0 -> 89.9 | Helped |
| Optimize camera trajectories with NeRF-based differentiable rendering and composition/joint losses instead of using the strongest baseline | Cinematic behavior transfer · PULL-OUT | IoU: 91.5 -> 94.0 | Helped |
| Optimize camera trajectories with NeRF-based differentiable rendering and composition/joint losses instead of using the strongest baseline | Cinematic behavior transfer · PAN | MPJPE: 40.9 -> 21.4 | Helped |
| Optimize camera trajectories with NeRF-based differentiable rendering and composition/joint losses instead of using the strongest baseline | Cinematic behavior transfer · TRACK | PA: 90.1 -> 94.5 | Helped |
| Optimize camera trajectories with NeRF-based differentiable rendering and composition/joint losses instead of using the strongest baseline | Cinematic behavior transfer · FOLLOW | MPJPE: 267.5 -> 130.9 | Helped |
| Optimize camera trajectories with NeRF-based differentiable rendering and composition/joint losses instead of the strongest baseline | Cinematic behavior transfer · ARC | IoU: 92.6 -> 94.5 | Helped |
Limitations
- Depends on SLAM-derived starting camera trajectories, failing when shot content changes too rapidly.
- Designed for shots prominently featuring humans; degrades to a SLAM-like method for environment-focused shots.
- Partially visible bodies in films lead to floating feet and inaccurate 3D placement.
Experimental setup
Datasets: film shots collected from the Internet (over 100 well-known shots)
Benchmarks: film shots collected from the Internet (over 100)
Metrics: Pixel Accuracy IoU MPJPE seven-point Likert user rating
Baselines: DROID-SLAM iNeRF JAWS PACE SLAHMR
Builds on: iNeRF SLAHMR D-NeRF JAWS PHALP
Closest papers by meaning
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.85)
- ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning (CVPR 2025 · similarity 0.85)
- Unveiling the Value of Motion for Cinematic Camera Trajectories (NeurIPS 2026 · similarity 0.85)
- Frame In-N-Out: Unbounded Controllable Image-to-Video Generation (NeurIPS 2025 · similarity 0.85)
- 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation (ICLR 2025 · similarity 0.84)
- Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.