Research map / 3D vision / Human pose estimation
You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains See3D, a pose-free visual-conditional multi-view diffusion model on a curated 16M-clip internet video dataset (WebVi3D) for open-world single-view and sparse-view 3D generation.
Problem
3D generation is limited by scarce, costly 3D datasets with camera pose annotations, so multi-view diffusion models cannot easily scale to open-world data.
Approach
Replace pose conditions with a purely 2D visual condition built by masking target frames, adding time-dependent noise, and mixing with the noisy latent over timesteps, enabling pose-free multi-view diffusion training on filtered static internet videos.
Key results
- Single-view generation on Tanks-and-Temples, RealEstate10K, CO3D achieves PSNR 23.76 / 25.36 / 24.28, beating ViewCrafter (re-implemented) by 4.63 dB average.
- With 3 sparse views on RealEstate10K, PSNR 28.04 vs CAT3D 22.02 and ReconFusion 20.74.
- Human annotation of 10,000 random clips found 88.6% (8,859) labeled as 3D-aware, validating the curation pipeline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use See3D instead of ViewCrafter | single-view generation · Tanks-and-Temples | PSNR: 19.66 -> 23.76 | Helped |
| Use See3D instead of ViewCrafter | single-view generation · RealEstate10K | PSNR: 21.93 -> 25.36 | Helped |
| Use See3D instead of ViewCrafter | single-view generation · CO3D | PSNR: 20.17 -> 24.28 | Helped |
| Use See3D instead of CAT3D | sparse-view reconstruction (3 views) · LLFF | PSNR: 21.58 -> 23.23 | Helped |
| Use See3D instead of CAT3D | sparse-view reconstruction (3 views) · DTU | PSNR: 22.02 -> 28.04 | Helped |
| Use See3D instead of CAT3D | sparse-view reconstruction (3 views) · MipNeRF-360 | PSNR: 16.62 -> 17.35 | Helped |
| Train on 10% rather than the full training set (ablation) | See3D · RealEstate10K | PSNR: 25.01 -> 19.32 | Hurt |
| Train on 20% rather than the full training set (ablation) | See3D · RealEstate10K | PSNR: 25.01 -> 21.04 | Hurt |
| Train on 40% rather than the full training set (ablation) | See3D · RealEstate10K | PSNR: 25.01 -> 22.57 | Hurt |
| Train on 80% rather than the full training set (ablation) | See3D · RealEstate10K | PSNR: 25.01 -> 24.08 | Hurt |
| Train on unfiltered rather than curated data (ablation) | See3D · RealEstate10K | PSNR: 25.01 -> 19.55 | Hurt |
| Use the full training set rather than 80% (ablation) | See3D · RealEstate10K | PSNR: 24.08 -> 25.01 | Helped |
Surprising findings
- Training on unfiltered video drops single-view PSNR to 19.55, indicating dynamic or low-motion content hurts multi-view consistency.
Limitations
- Relies on warping-based pipeline with monocular depth and keypoint alignment, which can accumulate errors over long sequences.
- Filtering pipeline uses several external models (segmentation, optical flow, tracking), adding complexity.
Experimental setup
Datasets: WebVi3D Tanks-and-Temples RealEstate10K CO3D LLFF DTU MipNeRF-360
Benchmarks: Tanks-and-Temples RealEstate10K CO3D LLFF DTU
Metrics: PSNR SSIM LPIPS
Baselines: LucidDreamer ZeroNVS MotionCtrl ViewCrafter MuRF FSGS BGGS DepthSplat ReconFusion CAT3D
Builds on: video diffusion model ViewCrafter CAT3D DDIM 3D Gaussian Splatting
Closest papers by meaning
- VideoMDM: Towards 3D Human Motion Generation From 2D Supervision (NeurIPS 2026 · similarity 0.83)
- Generating 3D-Consistent Videos from Unposed Internet Photos (CVPR 2025 · similarity 0.82)
- Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos (CVPR 2024 · similarity 0.82)
- Free3D: Consistent Novel View Synthesis without 3D Representation (CVPR 2024 · similarity 0.81)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.81)
- Cue3D: Quantifying the Role of Image Cues in Single-Image 3D Generation (NeurIPS 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.