Research map / Video / Video generation
ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
ReCapture re-renders a user-provided video along a new camera trajectory by generating a noisy anchor video (point-cloud rendering or per-frame multiview diffusion) and then regenerating it with masked video fine-tuning of spatial and temporal LoRAs on Stable Video Diffusion.
Problem
Generating new videos with user-specified camera trajectories from a single user-provided (non-generated) video that contains complex scene motion, without paired multi-view video training data.
Approach
Reformulates novel view synthesis of dynamic scenes as video-to-video translation: builds an incomplete anchor video via depth-based point cloud rendering or per-frame multiview diffusion, then fine-tunes a video diffusion model with a masked diffusion loss (temporal LoRA) and a context-aware spatial LoRA on source frames, so the model fills unknown regions while keeping scene motion.
Key results
- On Kubric-4D, ReCapture reaches PSNR 20.92 and SSIM 0.596 versus 20.30 and 0.587 for Generative Camera Dolly, without 4D training data.
- On VBench, ReCapture improves subject consistency from 80.42% to 92.02% and background consistency from 74.64% to 91.12% over Generative Camera Dolly.
- Each component helps in ablation: temporal LoRAs with masks raise subject consistency from 77.45% (anchor) to 90.88%, and SD-Edit lifts aesthetic quality to 82.07%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Generative Camera Dolly with ReCapture | video camera control · VBench | Temporal Smoothness: 82.33% -> 98.24% | Helped |
| replace Generative Camera Dolly with ReCapture | video camera control · VBench | Motion Degree: 51.24% -> 49.03% | Hurt |
| replace Generative Camera Dolly with ReCapture | novel view synthesis · Kubric-4D | PSNR (all): 20.30 -> 20.92 | Helped |
| replace Generative Camera Dolly with ReCapture | novel view synthesis · Kubric-4D | LPIPS (all): 0.408 -> 0.402 | Helped |
| add masked temporal LoRA fine-tuning (ablation) | ReCapture · VBench | Temporal Smoothness: 74.27% -> 97.32% | Helped |
| add masked temporal LoRA fine-tuning (ablation) | ReCapture · VBench | Subject Consistency: 77.45% -> 90.88% | Helped |
| add context-aware spatial LoRA (ablation) | ReCapture with temporal LoRA · VBench | Subject Consistency: 90.88% -> 91.24% | Helped |
| add context-aware spatial LoRA (ablation) | ReCapture with temporal LoRA · VBench | Temporal Smoothness: 97.32% -> 97.32% | No clear effect |
| apply SD-Edit after fine-tuning with both LoRAs (ablation) | ReCapture · VBench | Aesthetic Quality: 63.03% -> 64.75% | Helped |
| apply SD-Edit after fine-tuning with both LoRAs (ablation) | ReCapture · VBench | Dynamic Quality: 49.18% -> 57.35% | Helped |
Limitations
- Ill-posed problem: hallucinated content outside the original view is approximate and not verifiable.
- Point cloud rendering fails for large rotations, requiring the multiview diffusion alternative.
- Evaluation relies on a small VBench subset (35 videos) and a synthetic Kubric dataset.
Experimental setup
Datasets: Kubric-4D VBench
Benchmarks: Kubric-4D VBench
Metrics: PSNR SSIM LPIPS subject consistency background consistency temporal flickering motion smoothness dynamic degree aesthetic quality imaging quality
Baselines: Generative Camera Dolly HexPlane 4D-GS DynIBaR Vanilla SVD ZeroNVS
Builds on: Stable Video Diffusion CAT3D LoRA SDEdit Generative Camera Dolly
Closest papers by meaning
- Training-free Camera Control for Video Generation (ICLR 2025 · similarity 0.88)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.87)
- Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think (CVPR 2025 · similarity 0.87)
- Frame In-N-Out: Unbounded Controllable Image-to-Video Generation (NeurIPS 2025 · similarity 0.87)
- Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation (ICLR 2025 · similarity 0.86)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.