Research map / 3D vision / 3D generation
Generating 3D-Consistent Videos from Unposed Internet Photos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates camera-path videos interpolating between 2-5 unposed internet photos by jointly training a latent video DiT on multiview inpainting (MegaScenes) and view interpolation (RealEstate10k, DL3DV) without 3D annotations.
Problem
Generating consistent, realistic camera-trajectory videos between a handful of wide-baseline, unposed internet photos of the same scene; existing video models and frame interpolators hallucinate structure or cannot handle wide baselines.
Approach
Jointly train a latent video diffusion transformer with two self-supervised, 3D-aware objectives under one denoising loss: multiview inpainting of 80% masked target images from unordered internet photos, and view interpolation of video frames; plus CLIP-embedding illumination conditioning with color jitter augmentation.
Key results
- Ours (Full) is preferred over Luma in 60.21% (Phototourism) and 83.65% (Re10k) of consistency comparisons, and beats all frame interpolation baselines at 100% on Phototourism.
- COLMAP successfully reconstructs 235/252 sparse sets with generated frames vs 115/252 with only internet photos.
- InstantSplat PSNR on Phototourism improves from 11.70 (internet photos) to 13.96 on generated frames for Re10k-style scenes; Re10k PSNR rises from 19.86 to 21.80 on generated frames.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multiview inpainting training to video-only training (ablation) | KFC-W · Phototourism | SfM success rate: 179/252 -> 235/252 | Helped |
| Add multiview inpainting training to video-only training (ablation) | KFC-W · Phototourism | Consistency win rate: 76.55% | Helped |
| Use generated frames instead of sparse internet photos | InstantSplat · Phototourism | PSNR: 11.701 -> 13.960 | Helped |
| Use generated frames instead of original frames | InstantSplat · Re10k | PSNR: 19.857 -> 21.798 | Helped |
| Add generated views to sparse internet photos | COLMAP · Phototourism | Registered images: 378/882 -> 741/882 | Helped |
| Train only on view interpolation, without multiview inpainting or internet photos (ablation) | KFC-W · Phototourism | 3D consistency | Hurt |
| Train video-only with up to 5x wider-baseline inputs (ablation) | KFC-W · Phototourism | Scene layout consistency | Hurt |
| Remove CLIP embedding conditioning (ablation) | KFC-W · Video illumination consistency | illumination consistency | Hurt |
| Use modulation instead of addition for CLIP conditioning (ablation) | KFC-W · View interpolation | video quality | No clear effect |
| Normalize frame indices or use learnable positional encodings (ablation) | KFC-W · View interpolation | interpolation performance | No clear effect |
Surprising findings
- Video-only training with wide-baseline inputs (Long-Video ablation) fails on minimal-overlap inputs, showing internet photos contribute beyond wide-baseline video training.
- Commercial Luma was rated worse on Re10k indoor scenes than on Phototourism scenes, with artifacts such as disappearing objects.
Limitations
- Relies on an internal, non-public pretrained video model, limiting reproducibility.
- Does not model dynamic objects (people, vehicles), which are masked out and out of scope.
- Illumination control via CLIP embeddings is coarse; fine-grained lighting not controlled.
Experimental setup
Datasets: MegaScenes RealEstate10k DL3DV Phototourism
Benchmarks: Phototourism test set (252 sparse-view sets) RealEstate10k test set (600 sparse-view sets)
Metrics: user preference win rate (consistency, camera motion, aesthe COLMAP success rate registered images PSNR SSIM LPIPS
Baselines: Luma Dream Machine FILM FLAVR LDMVFI Ours (Video-only) InstantSplat
Builds on: CroCo DiT CogVideoX DDIM InstantSplat
Closest papers by meaning
- GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (CVPR 2025 · similarity 0.85)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.85)
- Scene Splatter: Momentum 3D Scene Generation from Single Image with Video Diffusion Model (CVPR 2025 · similarity 0.85)
- 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.85)
- 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation (CVPR 2024 · similarity 0.85)
- HiCoM: Hierarchical Coherent Motion for Dynamic Streamable Scenes with 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.