Research map / 3D vision / Depth and stereo
Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mines over 100K pseudo-metric 4D reconstructions (3D point clouds with long-term motion tracks) from internet VR180 stereo videos, and trains DynaDUSt3R, a DUSt3R variant with a motion head, on them.
Problem
Ground-truth 3D motion paired with real-world video is scarce; synthetic data fails to match real-world content and motion distributions, and motion capture or multi-view arrays are hard to scale.
Approach
Fuse camera pose estimation (SfM), per-frame RAFT stereo disparity, and BootsTAP 2D long-range tracks lifted to 3D, then refine per-track depth with a static/dynamic smoothness optimization, to produce large-scale real-world dynamic 3D data; add a time-conditioned motion head to DUSt3R to predict 3D scene flow to an arbitrary query time.
Key results
- DynaDUSt3R trained on Stereo4D reaches 3D EPE 0.1110 and 65.07% delta-0.05 on Stereo4D test, versus 0.6191 and 11.61% for the PointOdyssey-trained variant.
- On ADT, Stereo4D training gives EPE3D 0.1231 vs 0.3126 for PointOdyssey training.
- On Bonn depth, DynaDUSt3R (Stereo4D) reaches Abs Rel 0.0653 and delta<1.25 of 96.02, compared with 0.0721 / 92.60 for MonST3R and 0.1098 / 84.93 for DUSt3R.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train DynaDUSt3R on Stereo4D instead of PointOdyssey | DynaDUSt3R · Stereo4D test | EPE3D: 0.6191 -> 0.1110 | Helped |
| Train DynaDUSt3R on Stereo4D instead of PointOdyssey | DynaDUSt3R · Stereo4D test | δ0.05_3D: 11.61 -> 65.07 | Helped |
| Train DynaDUSt3R on Stereo4D instead of PointOdyssey | DynaDUSt3R · Stereo4D test | δ0.10_3D: 20.25 -> 75.18 | Helped |
| Train DynaDUSt3R on Stereo4D instead of PointOdyssey | DynaDUSt3R · ADT | EPE3D: 0.3126 -> 0.1231 | Helped |
| Train DynaDUSt3R on Stereo4D instead of PointOdyssey | DynaDUSt3R · ADT | δ0.05_3D: 8.56 -> 51.98 | Helped |
| Train DynaDUSt3R on Stereo4D instead of PointOdyssey | DynaDUSt3R · ADT | δ0.10_3D: 18.03 -> 65.20 | Helped |
| Train DynaDUSt3R on Stereo4D instead of PointOdyssey | DynaDUSt3R · Stereo4D test | Abs Rel: 0.3858 -> 0.1032 | Helped |
| Train DynaDUSt3R on Stereo4D instead of PointOdyssey | DynaDUSt3R · Stereo4D test | δ < 1.25: 61.87 -> 87.93 | Helped |
| Train DynaDUSt3R on Stereo4D instead of PointOdyssey | DynaDUSt3R · Bonn | Abs Rel: 0.0691 -> 0.0653 | Helped |
| Train DynaDUSt3R on Stereo4D instead of PointOdyssey | DynaDUSt3R · Bonn | δ < 1.25: 95.94 -> 96.02 | Helped |
Surprising findings
- The PointOdyssey-trained DynaDUSt3R produces non-zero motion on static elements such as building walls and street banners.
- Training on PointOdyssey gives worse depth on Stereo4D test (Abs Rel 0.3858) than the original DUSt3R (0.2696).
Limitations
- Long-range motion track quality depends on optical flow and 2D point tracking accuracy and degrades for distant background or long-occluded objects.
- DynaDUSt3R is non-generative and only handles two-frame inputs.
- Track optimization and filtering rely on heuristics such as semantic-category removal of drifting tracks.
Experimental setup
Datasets: Stereo4D PointOdyssey Bonn ADT (Aria Digital Twin) ADE20K
Benchmarks: Stereo4D held-out test set ADT (via TapVid3D) Bonn
Metrics: EPE3D delta 0.05 3D delta 0.10 3D Abs Rel delta < 1.25
Baselines: DUSt3R MonST3R DynaDUSt3R (PointOdyssey)
Builds on: DUSt3R BootsTAP RAFT COLMAP MonST3R
Closest papers by meaning
- MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (ICLR 2025 · similarity 0.90)
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.90)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.89)
- Flux4D: Flow-based Unsupervised 4D Reconstruction (NeurIPS 2025 · similarity 0.88)
- MUSt3R: Multi-view Network for Stereo 3D Reconstruction (CVPR 2025 · similarity 0.88)
- Fast Encoder-Based 3D from Casual Videos via Point Track Processing (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.