Research map / Detection and segmentation / Tracking
DriveTrack: A Benchmark for Long-Range Point Tracking in Real-World Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DriveTrack, a benchmark and automatic annotation pipeline that generates ~1B long-range point tracks on Waymo driving videos, and shows fine-tuning TAP trackers on it improves real-world tracking accuracy.
Problem
Long-range keypoint tracking (TAP) trackers are trained on synthetic data and perform poorly on real-world videos because there are almost no dense, accurate point-track annotations for real scenes.
Approach
Automatically annotate point tracks on real driving videos by shifting timestamped LiDAR point clouds according to ego-vehicle poses and object bounding-box transforms, with depth completion for occlusion estimation, yielding synthetic-benchmark-scale annotations for real footage.
Key results
- Fine-tuning TAP-Net on DriveTrack raises AJ on DriveTrack from 63.6% to 70.3% (+7%), and TAPIR from 78.8% to 84.1% (+5%).
- Fine-tuned TAPIR's AJ of 84.1% on DriveTrack is on par with TAPIR trained on Kubric (84.7% on Kubric).
- Fine-tuning on DriveTrack transfers to DAVIS, improving AJ by 1-2% across trackers (e.g., TAPIR 62.8 to 64.0 AJ on DAVIS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune TAP-Net on DriveTrack | TAP-Net · DriveTrack | AJ: 63.6 -> 70.3 | Helped |
| Fine-tune TAP-Net on DriveTrack | TAP-Net · DAVIS | AJ: 38.4 -> 39.2 | Helped |
| Fine-tune TAP-Net on DriveTrack | TAP-Net · Kubric | AJ: 65.4 -> 37.0 | Hurt |
| Fine-tune TAPIR on DriveTrack | TAPIR · DriveTrack | AJ: 78.8 -> 84.1 | Helped |
| Fine-tune TAPIR on DriveTrack | TAPIR · DAVIS | AJ: 62.8 -> 64.0 | Helped |
| Fine-tune TAPIR on DriveTrack | TAPIR · Kubric | AJ: 84.7 -> 80.8 | Hurt |
| Fine-tune PIPs++ on DriveTrack | PIPs++ · DriveTrack | <δx_avg: 81.5 -> 85.3 | Helped |
| Fine-tune PIPs++ on DriveTrack | PIPs++ · DAVIS | <δx_avg: 25.8 -> 25.6 | Hurt |
| Use Nearest Neighbor depth completion instead of CompletionFormer (ablation) | TAP-Net fine-tuned on DriveTrack · DAVIS | AJ: 38.60 -> 39.29 | Helped |
| Use Nearest Neighbor depth completion instead of CompletionFormer (ablation) | TAP-Net fine-tuned on DriveTrack · DriveTrack | AJ: 69.5 -> 70.3 | Helped |
Surprising findings
- Tracking error has a heavy tail whose 90th percentile stays above 60 pixels even after fine-tuning, suggesting more data alone will not fix tail performance.
- Fine-tuning TAP-Net and TAPIR on DriveTrack degrades their performance on the synthetic Kubric benchmark.
Limitations
- Annotations are restricted to rigid objects with bounding boxes, LiDAR point clouds and camera poses, limiting generalization to other domains.
- Occlusion and track quality depend on depth completion; nearest neighbor and CompletionFormer produce artifacts at object edges and thin structures.
- Objects beyond 20m or leaving the frame are filtered out, reducing coverage.
Experimental setup
Datasets: Waymo Open Dataset DAVIS Kubric PointOdyssey KITTI depth completion
Benchmarks: DriveTrack DAVIS Kubric
Metrics: AJ <δx avg OA
Baselines: TAP-Net TAPIR PIPs++ Harris corner SuperPoint CompletionFormer
Builds on: Track Any Point (TAP-Vid) TAPIR PIPs++ PointOdyssey
Closest papers by meaning
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.87)
- VastTrack: Vast Category Visual Object Tracking (NeurIPS 2024 · similarity 0.87)
- Is This Tracker On? A Benchmark Protocol for Dynamic Tracking (NeurIPS 2025 · similarity 0.87)
- TAPVid-3D: A Benchmark for Tracking Any Point in 3D (NeurIPS 2024 · similarity 0.86)
- M3SOT: Multi-Frame, Multi-Field, Multi-Space 3D Single Object Tracking (AAAI 2024 · similarity 0.85)
- Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.