Research map / 3D vision / 3D generation
Enhancing 3D Reconstruction for Dynamic Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D²USt3R, a feed-forward pointmap regression method that fine-tunes DUSt3R's decoder and DPT head to regress static-dynamic aligned pointmaps (SDAP) using occlusion/dynamic masks and a dynamic alignment loss, improving depth, pose, and dynamic correspondence on dynamic scenes.
Problem
DUSt3R-style pointmap regression assumes a static, rigid scene, so object motion misaligns pointmaps and corrupts correspondences and depth in dynamic scenes.
Approach
Extends pointmap regression with Static-Dynamic Aligned Pointmaps: static pixels are aligned by camera pose while dynamic pixels are aligned to their temporally corresponding positions in the other view via optical-flow correspondences, gated by occlusion and dynamic masks, plus optional flow and dynamic-mask heads.
Key results
- Multi-frame depth on TUM-Dynamics: AbsRel 0.142 and delta1 83.9 vs. MonST3R 0.145 / 81.2.
- Dynamic-region alignment EPE on Sintel-Clean: 16.19 vs. DUSt3R 30.96 and MonST3R 38.47; on KITTI 8.91 vs. 14.19 for DUSt3R.
- With an added flow head, EPE on KITTI is 3.57, outperforming SEA-RAFT (4.43).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use static-dynamic aligned pointmaps and motion-aware alignment training | D²USt3R · Dynamic pointmap alignment on Sintel-Clean, Sintel-Final, and KITTI | EPE: 30.96 -> 16.19; 35.11 -> 25.31; 14.19 -> 8.91 | Helped |
| Use D²USt3R instead of the strongest baseline | Multi-frame depth estimation · TUM-Dynamics and Bonn (all regions) | AbsRel: 0.145 -> 0.142; 0.068 -> 0.060 | Helped |
| Use D²USt3R instead of the strongest baseline | Multi-frame depth estimation · Sintel and KITTI (all regions) | AbsRel: 0.330 -> 0.324; 0.050 -> 0.104 | Mixed |
| Use D²USt3R instead of the strongest baseline | Single-frame depth estimation · Bonn, Sintel, KITTI, NYU-v2, and TUM-Dynamics | AbsRel: 0.076 -> 0.065; 0.345 -> 0.340; 0.076 -> 0.131; 0.080 -> 0.085; 0.147 -> 0.150 | Mixed |
| Add the optical flow head | D²USt3R · Sintel-Clean, Sintel-Final, and KITTI dynamic alignment | EPE: 16.19 -> 9.25; 25.31 -> 12.77; 8.91 -> 3.57 | Helped |
| Fine-tune the encoder as well as the decoder and head (ablation) | D²USt3R · TUM-Dynamics and Bonn | AbsRel: 0.142 -> 0.161; 0.060 -> 0.081 | Hurt |
| Add and jointly optimize the dynamic mask head (ablation) | D²USt3R · TUM-Dynamics, Bonn, Sintel, and KITTI | AbsRel: 0.142 -> 0.140; 0.060 -> 0.059; 0.324 -> 0.313; 0.104 -> 0.105 | Mixed |
Surprising findings
- Full fine-tuning including the encoder performs worse than fine-tuning only the decoder and head (TUM-Dynamics AbsRel 0.161 vs. 0.142).
- Performance on KITTI is limited, attributed to the absence of driving scenes in training data; comparable to MonST3R* rather than clearly superior.
Limitations
- Depends on the quality of the off-the-shelf optical flow model used for correspondences and masks.
- Non-generative and takes only two frames as input; extending to video requires global optimization.
- Weaker on KITTI driving scenes due to lack of such data in training.
Experimental setup
Datasets: BlinkVision Outdoor BlinkVision Indoor PointOdyssey TartanAir Spring
Benchmarks: TUM-Dynamics Bonn Sintel KITTI ScanNet NYU-v2 DAVIS
Metrics: AbsRel delta1 rotation error translation error end-point error
Baselines: DUSt3R MASt3R MonST3R MonST3R* DepthAnythingv2 Marigold Croco-Flow SEA-RAFT
Builds on: DUSt3R MonST3R RAFT ZeroCo Mixture-of-Laplace loss
Closest papers by meaning
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.89)
- MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (ICLR 2025 · similarity 0.88)
- ProDyG: Progressive Dynamic Scene Reconstruction via Gaussian Splatting from Monocular Videos (NeurIPS 2025 · similarity 0.87)
- Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory (NeurIPS 2025 · similarity 0.86)
- SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos (CVPR 2025 · similarity 0.86)
- Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.