Research map / 3D vision / Depth and stereo
Seurat: From Moving Points to Depth
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Seurat infers relative depth changes from 2D point trajectories using spatial and temporal transformers trained on synthetic Kubric data, then rescales them with an off-the-shelf monocular depth model.
Problem
Monocular video depth estimation is ambiguous and flickers over time; per-frame depth estimators lack temporal consistency and scale-consistent temporal depth changes.
Approach
Predict per-trajectory log depth ratios within sliding windows from 2D point tracks alone, using decoupled supporting (dense grid) and query trajectory transformer branches with cross-attention, then anchor the ratios to metric scale via a monocular depth estimator with piecewise median matching.
Key results
- On TAPVid-3D minival with per-trajectory scaling, Seurat (CoTracker) reaches 18.0 average APD (3D-AJ 25.1 on Aria) versus 13.4/15.0 range of depth-estimator baselines, e.g. CoTracker+DepthPro 9.7 3D-AJ avg.
- Seurat gives temporal coherence (TC) of 0.01 on Aria vs 0.06 for CoTracker+ZoeDepth, and more than 40x better coherence on DriveTrack than baselines.
- Removing the sliding window drops average 3D-AJ from 18.0 to 8.8 in ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the two-branch design for query and supporting trajectories (ablation) | Seurat · TAPVid-3D minival | 3D-AJ: 13.7 -> 18.0 | Helped |
| add sliding-window prediction (ablation) | Seurat · TAPVid-3D minival | 3D-AJ: 8.8 -> 18.0 | Helped |
| predict ratio depth relative to the window start instead of the query point (ablation) | Seurat · TAPVid-3D minival | 3D-AJ: 14.9 -> 18.0 | Helped |
| use 2 Transformer layers rather than 1 (ablation) | Seurat · TAPVid-3D minival | 3D-AJ: 15.5 -> 18.0 | Helped |
| use 3 Transformer layers rather than 2 (ablation) | Seurat · TAPVid-3D minival | 3D-AJ: 18.0 -> 17.8 | Hurt |
| use 4 Transformer layers rather than 2 (ablation) | Seurat · TAPVid-3D minival | 3D-AJ: 18.0 -> 17.8 | Hurt |
| add local RGB texture patches to the Transformer input (ablation) | Seurat · TAPVid-3D | 3D-AJ: 18.0 -> 15.8 | Hurt |
| apply 1 iteration of Gaussian smoothing to ZoeDepth + CoTracker (ablation) | ZoeDepth + CoTracker · TAPVid-3D, Aria | 3D-AJ: 9.8 -> 4.8 | Hurt |
| apply 3 iterations of Gaussian smoothing to ZoeDepth + CoTracker (ablation) | ZoeDepth + CoTracker · TAPVid-3D, Aria | 3D-AJ: 9.8 -> 4.4 | Hurt |
Surprising findings
- Adding RGB texture patches to the model degrades performance, suggested as overfitting to the synthetic training data.
- Gaussian smoothing of per-frame depth trajectories performs worse than Seurat, so gains are not merely from smoothing.
Limitations
- Depends on metric depth from an off-the-shelf MDE model for final scale, so absolute accuracy is bounded by that model.
- Relies on the quality of off-the-shelf point trackers and their occlusion predictions.
- Handcrafted density-based formulation is impractical; the learned model is needed, and extra layers beyond two did not help.
Experimental setup
Datasets: Kubric MOVi-F TAPVid-3D
Benchmarks: TAPVid-3D
Metrics: 3D-AJ APD TC
Baselines: CoTracker + ZoeDepth LocoTrack + ZoeDepth CoTracker + DepthPro LocoTrack + DepthPro DepthCrafter ChronoDepth TAPIR-3D
Builds on: CoTracker LocoTrack Kubric ZoeDepth
Closest papers by meaning
- Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth (NeurIPS 2026 · similarity 0.90)
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos (CVPR 2025 · similarity 0.89)
- TAPVid-360: Tracking Any Point in 360 from Narrow Field of View Video (NeurIPS 2025 · similarity 0.89)
- TAPVid-3D: A Benchmark for Tracking Any Point in 3D (NeurIPS 2024 · similarity 0.88)
- SpatialTracker: Tracking Any 2D Pixels in 3D Space (CVPR 2024 · similarity 0.88)
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.