Research map / Detection and segmentation / Tracking
Is This Tracker On? A Benchmark Protocol for Dynamic Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ITTO, a real-world point-tracking benchmark with long, occluded, multi-object tracks, and evaluates ten 2D and 3D trackers, finding large performance drops and poor re-identification after occlusion.
Problem
Existing point tracking benchmarks (TAP-Vid Davis/Kinetics, synthetic sets) are dominated by static, low-occlusion tracks and do not reveal why trackers fail under real-world motion, reappearance and occlusion.
Approach
A real-world benchmark built from MOSE, LVOS and Ego4D videos with algorithmically sampled query points and a two-phase human annotation pipeline (MTurk coarse tracks, then expert refinement), plus metrics partitioned by track motion and reappearance count.
Key results
- CoTracker3 achieves 64.4 AJ on TAP-Vid DAVIS but only 42.0 on ITTO (reported per-table as 35.3 AJ online on ITTO), indicating large degradation.
- On high-motion tracks (>5% diagonal), CoTracker3 AJ falls from 55.9 to 13.7; on tracks with 3+ reappearances AJ falls from 50.1 to 13.7.
- Only 30.7% of ITTO tracks are static versus 82.3% in DAVIS and 88.1% in Kinetics; ITTO has 5.86 reappearances per track versus 0.10 in DAVIS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate CoTracker3 on ITTO instead of TAP-Vid Davis | CoTracker3 · ITTO vs. TAP-Vid Davis | AJ: 64.4 -> 35.5 | Hurt |
| Evaluate CoTracker3 on ITTO instead of TAP-Vid Kinetics | CoTracker3 · ITTO vs. TAP-Vid Kinetics | AJ: 54.7 -> 35.5 | Hurt |
| Track higher frame-to-frame motion (from [0%,0.5%) to [5%,100%)) | CoTracker3 online · ITTO | AJ: 55.9 -> 13.7 | Hurt |
| Track more frequent reappearances (from [0,1) to [3,1000)) | CoTracker3 online · ITTO | AJ: 50.1 -> 13.7 | Hurt |
| Increase track occlusion rate from (0,24] to (72,100] | CoTracker3 · ITTO | AJ: 43.9 -> 24.7 | Hurt |
| Track complex object motion (PDV ≥ 0.05 rather than PDV < 0.05) | CoTracker3 · ITTO | AJ: 45.2 -> 29.5 | Hurt |
| Use background rather than gradient-based query points | CoTracker3 online · ITTO | AJ: 33.2 -> 39.1 | Helped |
| Use background rather than random query points | CoTracker3 online · ITTO | AJ: 34.3 -> 39.1 | Helped |
| Add negative prompts when generating SAM2 pseudomasks | Ego4D pseudomask pipeline · ITTO annotation masks | mask quality | Helped |
Surprising findings
- Occlusion accuracy is higher for high-occlusion tracks than low-occlusion tracks, so OA alone is an insufficient diagnostic.
- Trackers rank differently on ITTO than on TAP-Vid; e.g., BootsTAP drops about 15% relative to CoTracker3 despite reporting higher AJ on TAP-Vid.
Limitations
- Human annotations limit precision to the pixel level, which may matter for high-precision domains such as surgery.
- Featureless regions such as sky and water are omitted.
- Domain coverage is limited; many environments are not represented.
Experimental setup
Datasets: MOSE LVOS Ego4D TAP-Vid DAVIS TAP-Vid Kinetics Kubric
Benchmarks: ITTO TAP-Vid DAVIS TAP-Vid Kinetics
Metrics: Average Jaccard delta (points within threshold) occlusion accuracy pairwise distance variance
Baselines: CoTracker3 LocoTrack TAPIR BootsTAP TAPTR TAPNext SpatialTracker SceneTracker DELTA
Builds on: TAP-Vid SAM2 Molmo
Closest papers by meaning
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.88)
- DriveTrack: A Benchmark for Long-Range Point Tracking in Real-World Videos (CVPR 2024 · similarity 0.87)
- TAPVid-3D: A Benchmark for Tracking Any Point in 3D (NeurIPS 2024 · similarity 0.86)
- ETAP: Event-based Tracking of Any Point (CVPR 2025 · similarity 0.86)
- TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation (NeurIPS 2024 · similarity 0.86)
- LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.