Research map / Point clouds / Point cloud classification
TAPVid-3D: A Benchmark for Tracking Any Point in 3D
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TAPVid-3D, a real-world benchmark of 4,000+ clips for long-range 3D point tracking built from Aria Digital Twin, DriveTrack and Panoptic Studio, with new 3D-AJ/APD/OA metrics and baseline evaluations of 2D trackers combined with depth estimators.
Problem
Existing real-world tracking-any-point benchmarks only evaluate 2D tracking because they lack metric 3D ground-truth trajectories, so 3D point tracking from monocular video cannot be measured on real footage.
Approach
Derive metric 3D point trajectories and visibility from three existing real-world datasets (digital-twin annotations, LIDAR plus 3D boxes, and dynamic Gaussian splat reconstructions), and extend the TAP Average Jaccard metric to 3D with depth-adaptive thresholds and several scale-alignment schemes.
Key results
- BootsTAPIR + ZoeDepth achieves 8.8 average 3D-AJ on the full_eval split with median scaling, while the best average 3D-AJ overall is 9.3 (CoTracker + COLMAP and BootsTAPIR + COLMAP).
- SpatialTracker reaches the best average APD of 15.5 under median scaling, and 2D tracking on TAPVid-3D projections reaches 59.1 2D-AJ for BootsTAPIR.
- With per-trajectory scaling, BootsTAPIR + COLMAP reaches 20.6 average 3D-AJ, versus 9.3 under global median scaling, showing scale consistency is a major bottleneck.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COLMAP instead of ZoeDepth for depth estimation | 2D point trackers on TAPVid-3D DriveTrack · DriveTrack | 3D-AJ: 5.4 -> 11.8 | Helped |
| Use ZoeDepth instead of COLMAP for depth estimation | 2D point trackers on TAPVid-3D Panoptic Studio · Panoptic Studio | 3D-AJ: 6.9 -> 11.3 | Helped |
| Use BootsTAPIR instead of TAPIR with ZoeDepth | 2D point tracking plus depth estimation · TAPVid-3D full_eval (Average) | 3D-AJ: 8.3 -> 8.8 | Helped |
| Use global median scaling instead of per-trajectory scaling (ablation) | BootsTAPIR + COLMAP · TAPVid-3D full_eval (Average) | 3D-AJ: 9.3 -> 20.6 | Hurt |
| Use global median scaling instead of local neighborhood scaling (ablation) | BootsTAPIR + COLMAP · TAPVid-3D full_eval (Average) | 3D-AJ: 9.3 -> 17.8 | Hurt |
| Use per-trajectory scaling instead of global median scaling (ablation) | TAPIR-3D · TAPVid-3D full_eval (Average) | 3D-AJ: 3.1 -> 8.6 | Helped |
| Use local neighborhood scaling instead of global median scaling (ablation) | TAPIR-3D · TAPVid-3D full_eval (Average) | 3D-AJ: 3.1 -> 6.1 | Helped |
| Use per-trajectory scaling instead of global median scaling (ablation) | BootsTAPIR + ZoeDepth · TAPVid-3D full_eval (Average) | 3D-AJ: 8.8 -> 12.3 | Helped |
Surprising findings
- Baseline 2D tracking looks accurate while 3D tracking performance is significantly lower, with the static baseline reaching competitive occlusion accuracy.
- TAPIR-3D, trained only on synthetic Kubric data, gives competitive per-trajectory results on DriveTrack, outperforming SpatialTracker there.
Limitations
- Only solid opaque objects are evaluated, inheriting TAP and monocular depth limitations.
- Automatic annotations may be imperfect due to sensor noise, and the three sources do not cover all domains.
- Dataset licenses are non-commercial, and it may inherit biases from lab participants and six US cities.
Experimental setup
Datasets: Aria Digital Twin DriveTrack Panoptic Studio Waymo Open Dataset Kubric
Benchmarks: TAPVid-3D TAPVid-DAVIS
Metrics: 3D-AJ APD OA 2D-AJ
Baselines: TAPIR + COLMAP CoTracker + COLMAP BootsTAPIR + COLMAP TAPIR + ZoeDepth CoTracker + ZoeDepth BootsTAPIR + ZoeDepth TAPIR-3D SpatialTracker Static Baseline
Builds on: TAP-Vid TAPIR BootsTAPIR CoTracker COLMAP ZoeDepth SpatialTracker
Closest papers by meaning
- TAPIP3D: Tracking Any Point in Persistent 3D Geometry (NeurIPS 2025 · similarity 0.94)
- TAPVid-360: Tracking Any Point in 360 from Narrow Field of View Video (NeurIPS 2025 · similarity 0.90)
- TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels (NeurIPS 2025 · similarity 0.89)
- DELTA: DENSE EFFICIENT LONG-RANGE 3D TRACKING FOR ANY VIDEO (ICLR 2025 · similarity 0.88)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.88)
- Seurat: From Moving Points to Depth (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.