Research map / 3D vision / Depth and stereo
TAPIP3D: Tracking Any Point in Persistent 3D Geometry
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAPIP3D, a feed-forward long-term 3D point tracker that lifts RGB-D video features into a camera-stabilized world-space 3D feature cloud and uses 3D neighborhood-to-neighborhood attention to iteratively refine trajectories.
Problem
Long-term point tracking in monocular RGB and RGB-D videos is usually done in pixel or UVD (image plane plus depth) space, where camera motion confounds object motion, making 3D trajectories less smooth and harder to predict.
Approach
Represent video as spatio-temporal 3D feature clouds unprojected from depth and transformed into a first-camera world frame to cancel camera motion, then replace 2D square correlation windows with 3D k-NN neighborhood-to-neighborhood cross-attention inside a CoTracker3-style iterative transformer.
Key results
- On TAPVid-3D with MegaSaM depth and poses, TAPIP3D-world reaches 18.8 average AJ3D versus 17.8 for DELTA and 17.3 for CoTracker3.
- On LSFOdyssey with GT depth, TAPIP3D-world reaches 72.2 AJ3D versus 37.7 for DELTA.
- On DexYCB-Pt with sensor depth, AJ3D improves from 26.4 (DELTA) to 30.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| track in world coordinates rather than camera coordinates | TAPIP3D · TAPVid-3D average | AJ_3D: 18.1 -> 18.8 | Helped |
| track in world coordinates rather than camera coordinates (ablation) | TAPIP3D · LSFOdyssey, XYZ coordinates | AJ_3D: 67.1 -> 70.7 | Helped |
| use XYZ world coordinates instead of UV + D (ablation) | TAPIP3D · LSFOdyssey | AJ_3D: 63.4 -> 70.7 | Helped |
| use XYZ world coordinates instead of UV + log(D) (ablation) | TAPIP3D · LSFOdyssey | AJ_3D: 62.9 -> 70.7 | Helped |
| add N2N attention (ablation) | TAPIP3D-camera · LSFOdyssey | AJ_3D: 59.4 -> 67.1 | Helped |
| add N2N attention (ablation) | TAPIP3D-world · LSFOdyssey | APD_3D: 75.1 -> 84.1 | Helped |
| use 3D k-NN instead of fixed 2D neighbors (ablation) | TAPIP3D · DexYCB-Pt, sensor depth | AJ_3D: 27.7 -> 29.8 | Helped |
| increase neighborhood size from K=16 to K=32 (ablation) | TAPIP3D · LSFOdyssey | AJ_3D: 67.9 -> 72.2 | Helped |
| increase neighborhood size from K=32 to K=48 (ablation) | TAPIP3D · LSFOdyssey | AJ_3D: 72.2 -> 72.6 | Helped |
| improve depth from MegaSaM estimates to ground-truth depth | TAPIP3D-world · LSFOdyssey, AJ_3D | AJ_3D: 20.5 -> 72.2 | Helped |
| track in 3D and project trajectories to image coordinates | TAPIP3D-world · TAP-Vid-DAVIS | AJ | Hurt |
| apply Gaussian blur and re-estimate depth | TAPIP3D-world · LSFOdyssey, APD_3D | APD_3D: 29.2 -> 18.1 | Hurt |
Surprising findings
- On TAP-Vid-DAVIS, TAPIP3D-world is lower than recent state-of-the-art 2D trackers, and projected 2D tracks cannot reach sub-pixel accuracy in monocular settings.
- On LSFOdyssey with MegaSaM depth, TAPIP3D-world's 2D AJ is below CoTracker3 and DELTA (72.3 vs 75.0 and 75.2).
Limitations
- Performance depends on input depth fidelity; MegaSaM failures on extreme depth variation or small distant objects degrade tracking.
- Without high-quality depth, 2D metrics can be worse than UVD-space or purely 2D trackers.
- N2N attention is costly, O(TNK^2D + TNKD^2), making it a computational bottleneck.
Experimental setup
Datasets: Kubric MOVi-F TAPVid-3D LSFOdyssey Dynamic Replica DexYCB
Benchmarks: TAPVid-3D DexYCB-Pt LSFOdyssey Dynamic Replica TAP-Vid-DAVIS
Metrics: AJ3D APD3D OA AJ2D APD2D
Baselines: CoTracker3 SpatialTracker DELTA PIPs
Builds on: CoTracker3 PIPs TAPVid-3D MegaSaM LocoTrack
Closest papers by meaning
- TAPVid-3D: A Benchmark for Tracking Any Point in 3D (NeurIPS 2024 · similarity 0.94)
- TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels (NeurIPS 2025 · similarity 0.89)
- TAPVid-360: Tracking Any Point in 360 from Narrow Field of View Video (NeurIPS 2025 · similarity 0.89)
- DELTA: DENSE EFFICIENT LONG-RANGE 3D TRACKING FOR ANY VIDEO (ICLR 2025 · similarity 0.88)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.87)
- SpatialTracker: Tracking Any 2D Pixels in 3D Space (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.