Research map / 3D vision / 3D generation
Continuous 3D Perception Model with Persistent State
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CUT3R, a stateful recurrent transformer that updates a persistent state with each image to output metric pointmaps and camera poses online, and can query unseen views via raymaps.
Problem
Online dense 3D reconstruction and camera pose estimation from streams of images or unordered photo collections, handling both static and dynamic scenes and inferring unobserved regions, without known intrinsics or poses.
Approach
A persistent learnable state tokens updated via bidirectional transformer decoders (state-update and state-readout) with each image, predicting pointmaps in self and world frames plus ego pose; unseen structure is inferred by querying the state with a raymap of a virtual camera.
Key results
- On video depth with per-sequence scale, Ours achieves Bonn Abs Rel 0.078 and KITTI Abs Rel 0.118, outperforming online Spann3R and DUSt3R-GA on those datasets while running at 16.58 FPS on KITTI.
- On 7-Scenes 3D reconstruction, Ours reaches mean accuracy 0.126 vs 0.146 for DUSt3R-GA, while running online at about 25x the speed.
- Revisiting the final state improves NRGBD accuracy from 0.099 to 0.094 mean accuracy, showing the state refines with more observations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CUT3R instead of the best listed baseline | Monocular depth estimation · Bonn | Abs Rel: 0.076 -> 0.063 | Helped |
| Use CUT3R instead of the best listed baseline | Monocular depth estimation · KITTI | Abs Rel: 0.079 -> 0.092 | Hurt |
| Use CUT3R instead of the best listed baseline | Video depth estimation · Sintel, per-sequence scale | Abs Rel: 0.378 -> 0.421 | Hurt |
| Use CUT3R instead of MASt3R-GA | Video depth estimation · Sintel, metric scale | Abs Rel: 1.022 -> 1.029 | Hurt |
| Use CUT3R instead of MonST3R-GA | Video depth estimation · Bonn, per-sequence scale | Abs Rel: 0.067 -> 0.078 | Hurt |
| Use CUT3R instead of the best listed baseline | Video depth estimation · KITTI, per-sequence scale | Abs Rel: 0.168 -> 0.118 | Helped |
| Use CUT3R instead of the best listed baseline | Camera pose estimation · Sintel | ATE: 0.111 -> 0.213 | Hurt |
| Use CUT3R instead of MASt3R-GA | Camera pose estimation · TUM-dynamics | ATE: 0.038 -> 0.046 | Hurt |
| Use CUT3R instead of MonST3R-GA | Camera pose estimation · ScanNet | ATE: 0.077 -> 0.099 | Hurt |
| Use CUT3R instead of DUSt3R-GA | 3D reconstruction · 7-Scenes | Acc: 0.146 -> 0.126 | Helped |
| Use CUT3R instead of MASt3R-GA | 3D reconstruction · NRGBD | Acc: 0.085 -> 0.099 | Hurt |
| Revisit images using the final state (ablation) | CUT3R · 7-Scenes | Acc: 0.126 -> 0.113 | Helped |
Surprising findings
- On Sintel single-frame depth, the method is worse than DUSt3R and MASt3R (Abs Rel 0.428 vs 0.340 for MASt3R), and its Sintel pose ATE is worse than the best optimization-based methods.
Limitations
- May drift over very long sequences due to lack of global alignment.
- Deterministic raymap-based structure generation yields blurry results when extrapolating far from input views.
- Training recurrent networks is time-consuming.
Experimental setup
Datasets: CO3Dv2 ARKitScenes ScanNet++ TartanAir Waymo MegaDepth MapFree DL3DV DynamicStereo RealEstate10K Synscapes
Benchmarks: KITTI Sintel Bonn NYU-v2 TUM-dynamics ScanNet 7-Scenes NRGBD
Metrics: Abs Rel delta<1.25 ATE RPE trans RPE rot Acc Comp NC FPS
Baselines: DUSt3R MASt3R Spann3R MonST3R Robust-CVD CasualSAM Particle-SfM
Builds on: DUSt3R MASt3R DPT
Closest papers by meaning
- BAT3R: Robust Online 3D Reconstruction with Bayesian Adaptive State Updates (NeurIPS 2026 · similarity 0.88)
- SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images (CVPR 2025 · similarity 0.88)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.87)
- Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory (NeurIPS 2025 · similarity 0.87)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.87)
- Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.