Research map / 3D vision / Depth and stereo
SpatialTracker: Tracking Any 2D Pixels in 3D Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpatialTracker, which lifts 2D pixels to 3D using monocular depth, encodes frames as triplanes, and iteratively predicts 3D point trajectories with a transformer regularized by a learned-rigidity ARAP loss.
Problem
Long-range dense pixel tracking in videos is hard due to 3D-to-2D projection causing occlusions and discontinuities; existing point trackers correlate features only in 2D image space.
Approach
Lift pixels to 3D with monocular depth, represent each frame as a triplane feature representation, track points iteratively in 3D with a transformer, and enforce an as-rigid-as-possible constraint using a self-supervised rigidity embedding that groups pixels into rigid parts.
Key results
- On TAP-Vid average, achieves AJ 58.2, <δavg 73.3, OA 88.2, outperforming CoTracker (57.4 AJ).
- On PointOdyssey 3D tracking, ATE3D 0.22 vs 0.77 for lifted CoTracker and 0.70 for chained RAFT-3D.
- Ablation on DAVIS shows removing ARAP drops AJ from 61.1 to 55.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Track in 3D using triplane features and ARAP regularization rather than the strongest listed 2D baseline | SpatialTracker · TAP-Vid Kinetics, DAVIS, and RGB-Stacking | AJ: 57.4 -> 58.2 | Helped |
| Track in 3D using triplane features and ARAP regularization rather than the strongest listed 2D baseline | SpatialTracker · BADJA | segA: 63.6 -> 69.2 | Helped |
| Track in 3D using triplane features and ARAP regularization rather than CoTracker | SpatialTracker · PointOdyssey | MTE: 30.5 -> 28.3 | Helped |
| Track in 3D rather than chain pairwise scene flow | SpatialTracker · PointOdyssey 3D | ATE3D: 0.70 -> 0.22 | Helped |
| Add the ARAP loss (ablation) | SpatialTracker · TAP-Vid DAVIS | AJ: 55.1 -> 61.1 | Helped |
| Replace ZoeDepth with DPT (ablation) | SpatialTracker · TAP-Vid DAVIS | AJ: 61.1 -> 51.4 | Hurt |
| Replace ZoeDepth with MiDaS (ablation) | SpatialTracker · TAP-Vid DAVIS | AJ: 61.1 -> 56.3 | Hurt |
| Replace ZoeDepth with MiDaS (ablation) | SpatialTracker · TAP-Vid DAVIS | <δavg: 76.3 -> 73.9 | Hurt |
| Replace ZoeDepth with DPT (ablation) | SpatialTracker · TAP-Vid DAVIS | OA: 89.5 -> 83.3 | Hurt |
Surprising findings
- OmniMotion, which also tracks in 3D, outperforms SpatialTracker on TAP-Vid DAVIS and RGB-Stacking, but requires costly test-time optimization.
- On BADJA, SpatialTracker's segA (69.2) is worse than the best baseline's segA in the table ordering (lower is better), though its δ3px is competitive.
Limitations
- Performance depends on accuracy of off-the-shelf monocular depth estimators.
- Training relies on synthetic Kubric data with ground-truth 3D trajectories.
Experimental setup
Datasets: TAP-Vid-Kubric
Benchmarks: TAP-Vid (DAVIS, Kinetics, RGB-Stacking) BADJA PointOdyssey
Metrics: AJ average position accuracy (<δx avg) occlusion accuracy segA δ3px MTE survival rate ATE3D δ0.1 δ0.2
Baselines: TAP-Net PIPs OmniMotion TAPIR CoTracker RAFT-3D
Builds on: CoTracker TAP-Net PIPs RAFT-3D
Closest papers by meaning
- TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels (NeurIPS 2025 · similarity 0.89)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.88)
- Seurat: From Moving Points to Depth (CVPR 2025 · similarity 0.88)
- TAPIP3D: Tracking Any Point in Persistent 3D Geometry (NeurIPS 2025 · similarity 0.87)
- DELTA: DENSE EFFICIENT LONG-RANGE 3D TRACKING FOR ANY VIDEO (ICLR 2025 · similarity 0.87)
- TAPVid-3D: A Benchmark for Tracking Any Point in 3D (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.