Research map / Detection and segmentation / Tracking
TAPNext: Tracking Any Point (TAP) as Next Token Prediction
Artem Zholus, Carl Doersch, Yi Yang, Skanda Koppula, Viorica Patraucean, Xu Owen He, Ignacio Rocco, Mehdi S. M. Sajjadi, Sarath Chandar, Ross Goroshin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use classification coordinate head instead of regression head (ablation) | TAPNext-S · DAVIS query-first (small-scale ablation) | average Jaccard: 44.7 -> 55.0 | Improved |
| add classification loss to regression coordinate loss (ablation) | TAPNext-S · DAVIS query-first (small-scale ablation) | average Jaccard: 48.1 -> 55.0 | Improved |
| add regression loss to classification-only coordinate loss (ablation) | TAPNext-S · DAVIS query-first (small-scale ablation) | average Jaccard: 52.7 -> 55.0 | Improved |
| use 8x8 image patch instead of 16x16 (ablation) | TAPNext-S · DAVIS query-first (small-scale ablation) | average Jaccard: 49.7 -> 55.0 | Improved |
| add losses after each ViT block (ablation) | TAPNext-S · DAVIS query-first (small-scale ablation) | average Jaccard: 50.5 -> 55.0 | Improved |
| add 2x state expansion in the SSM (ablation) | TAPNext-S · DAVIS query-first (small-scale ablation) | average Jaccard: 53.8 -> 55.0 | Improved |
| use temporal SSM instead of temporal attention (ablation) | TAPNext-S · DAVIS full-length videos | average Jaccard: 17.3 -> 55.0 | Improved |
| use temporal SSM instead of temporal attention (ablation) | TAPNext-S · DAVIS 24-frame videos | average Jaccard: 68.4 -> 70.0 | Improved |
| track points jointly instead of one at a time with support grids (ablation) | TAPNext-B · DAVIS first | average Jaccard: 62.2 -> 62.4 | No clear effect |
| add image reconstruction objective on output image tokens (ablation) | TAPNext · TAP-Vid | tracking performance | No clear effect |
| add BootsTAP-style self-training on real videos | TAPNext-B · DAVIS first | average Jaccard: 62.4 -> 65.2 | Improved |
| TAPNext online vs CoTracker3 online latency (256 queries, H100) | TAPNext-B · latency | latency ms: 80 -> 5.05 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.
- TAPIR — TAPIR: Tracking Any Point with Per-Frame Initialization and Temporal Refinement (ICCV 2023) · better on 4 of 4 shared results
- Track-On — Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025) · better on 2 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- TAPTRv2: Attention-based Position Update Improves Tracking Any Point (NeurIPS 2024 · similarity 0.88)
- TAPVid-3D: A Benchmark for Tracking Any Point in 3D (NeurIPS 2024 · similarity 0.87)
- ETAP: Event-based Tracking of Any Point (CVPR 2025 · similarity 0.87)
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.87)
- TAPVid-360: Tracking Any Point in 360 from Narrow Field of View Video (NeurIPS 2025 · similarity 0.86)
- ODTrack: Online Dense Temporal Token Learning for Visual Tracking (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
