Research map / 3D vision / Depth and stereo
Puzzles: Unbounded Video-Depth Augmentation for Scalable End-to-End 3D Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Puzzles is a plug-and-play data augmentation that synthesizes posed video-depth clips from single RGB-D images (Image-to-Clips) or keyframe-selected video clips (Clips-to-Clips) to train video-based 3R-series 3D reconstruction models.
Problem
Feedforward end-to-end 3D reconstruction models (DUSt3R-successors such as Spann3R, SLAM3R, Fast3R) depend on large volumes of posed video-depth data, which are limited in diversity and scale.
Approach
Create ordered, overlapping patches from a single RGB-D image, calibrate per-patch intrinsics/extrinsics (PnP with RANSAC), apply centroid-based random rotations with validity checks to form pseudo-video clips; extend to videos by overlap-matrix-based keyframe selection before augmentation.
Key results
- Fast3R with Puzzles (full data) reduces 7Scenes Acc from 0.0412 to 0.0342 (+16.99%) and Comp by 13.09%.
- Spann3R trained with Puzzles on 1/10 of data reaches DTU Acc 4.9832 vs 6.2432 for full-data baseline (+20.18%).
- SLAM3R with Puzzles (full data) improves DTU Acc by 16.71% and Comp by 16.12%; on NRGBD gains are 6-10% with full data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Puzzles augmentation with full training data | Spann3R · 7Scenes | Acc: 0.0388 -> 0.0330 | Helped |
| Add Puzzles augmentation with full training data | Spann3R · NRGBD | Acc: 0.0686 -> 0.0644 | Helped |
| Add Puzzles augmentation with full training data | Fast3R · DTU | Acc: 4.2961 -> 3.5912 | Helped |
| Add Puzzles augmentation with full training data | SLAM3R · 7Scenes | Acc: 0.0291 -> 0.0264 | Helped |
| Add Puzzles augmentation with 1/10 training data | Fast3R · NRGBD | Acc: 0.0735 -> 0.0772 | Hurt |
| Add Puzzles augmentation with 1/10 training data | Spann3R · 7Scenes | Acc: 0.0388 -> 0.0389 | No clear effect |
| Add Puzzles augmentation with 1/10 training data | SLAM3R · NRGBD | Comp: 0.0292 -> 0.0313 | Hurt |
| Add Puzzles augmentation with 1/10 training data | Spann3R · DTU | Acc: 6.2432 -> 4.9832 | Helped |
| Add Puzzles augmentation with full training data | Fast3R · 7Scenes | Comp: 0.0275 -> 0.0239 | Helped |
| Add Puzzles augmentation with full training data | SLAM3R · NRGBD | Comp: 0.0292 -> 0.0263 | Helped |
| Add centroid-based camera rotation to translation-only augmentation (ablation) | Spann3R with Image-to-Clips · DTU | Acc: 0.0508 -> 0.0330 | Helped |
| Use predicted rather than ground-truth depth for Image-to-Clips (ablation) | Spann3R with Image-to-Clips · DTU | Acc: 5.1 -> 4.9 | No clear effect |
Surprising findings
- On NRGBD, models trained on 1/10 of the data with Puzzles do not match full-data baselines, unlike on 7Scenes.
- Predicted monocular depth for Image-to-Clips gives marginally better DTU accuracy (about 4.9) than ground-truth depth (5.1).
Limitations
- Assumes pinhole camera model and static scenes.
- 2D augmentations do not update camera parameters.
- Image-to-Clips drifts on long scene-level sequences; augmentation is scene-specific rather than cross-scene.
Experimental setup
Datasets: ScanNet-v2 ARKitScenes Habitat BlendedMVS
Benchmarks: 7Scenes NRGBD DTU
Metrics: Accuracy Completion
Baselines: Spann3R Fast3R SLAM3R
Builds on: DUSt3R Spann3R SLAM3R Fast3R Open3D
Closest papers by meaning
- Test3R: Learning to Reconstruct 3D at Test Time (NeurIPS 2025 · similarity 0.86)
- DUSt3R: Geometric 3D Vision Made Easy (CVPR 2024 · similarity 0.85)
- DepthSplat: Connecting Gaussian Splatting and Depth (CVPR 2025 · similarity 0.85)
- MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds (CVPR 2025 · similarity 0.85)
- Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos (CVPR 2025 · similarity 0.85)
- MUSt3R: Multi-view Network for Stereo 3D Reconstruction (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.