Research map / Architectures and efficiency / Attention and transformers
TAPTRv2: Attention-based Position Update Improves Tracking Any Point
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAPTRv2, a DETR-style tracking-any-point model that replaces cost-volume aggregation with an attention-based position update (key-aware deformable attention) to keep point query content features uncontaminated.
Problem
Tracking any point (TAP) in videos; TAPTR's cost-volume aggregation contaminates point query content features, hurting visibility prediction, attention computation, and cost-volume accuracy, and adds redundant computation.
Approach
Attention-based position update: local attention weights (computed by dot product between query and sampled features, like cost-volume) are used to combine sampling offsets into a position update, while content is updated separately; a disentangler MLP separates the attention weights for content and position updates, and extra per-layer position supervision is added.
Key results
- On TAP-Vid-DAVIS (First mode), TAPTRv2 reaches 63.5 AJ, 75.9 <delta_avg, 91.4 OA, versus TAPTR's 63.0 AJ and BootsTAP's 61.4 AJ.
- On TAP-Vid-Kinetics, TAPTRv2 reports 49.7 AJ versus TAPTR's 49.0 AJ and TAPIR's 49.6 AJ.
- Ablation: key-aware attention adds 0.7 AJ, attention-based position update 1.0 AJ, weight disentangling 0.9 AJ, and extra supervision 0.9 AJ on the 5-layer model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace TAPTR with TAPTRv2's attention-based position update and key-aware deformable attention | Point tracking · DAVIS | AJ: 63.0 -> 63.5 | Helped |
| Replace TAPTR with TAPTRv2's attention-based position update and key-aware deformable attention | Point tracking · DAVIS-S | AJ: 66.3 -> 66.4 | Helped |
| Replace TAPTR with TAPTRv2's attention-based position update and key-aware deformable attention | Point tracking · Kinetics | AJ: 49.0 -> 49.7 | Helped |
| Add self-attention (ablation) | TAPTR decoder · DAVIS (out of domain) | AJ: 47.4 -> 50.6 | Helped |
| Add self-attention (ablation) | TAPTR decoder · Kubric (in domain) | AJ: 79.7 -> 83.7 | Helped |
| Add temporal attention (ablation) | TAPTR decoder · DAVIS (out of domain) | AJ: 47.4 -> 54.3 | Helped |
| Add temporal attention (ablation) | TAPTR decoder · Kubric (in domain) | AJ: 79.7 -> 83.4 | Helped |
| Add cost volume (ablation) | TAPTR decoder · DAVIS (out of domain) | AJ: 47.4 -> 52.0 | Helped |
| Add cost volume (ablation) | TAPTR decoder · Kubric (in domain) | AJ: 79.7 -> 79.5 | Hurt |
| Add key-aware deformable attention (ablation) | TAPTRv2 · DAVIS | AJ: 60.0 -> 60.7 (+0.7 AJ) | Helped |
| Add attention-based position update (ablation) | TAPTRv2 with key-aware attention · DAVIS | AJ: 60.7 -> 61.7 (1.0 AJ) | Helped |
| Add attention-weight disentangling (ablation) | TAPTRv2 with position update · DAVIS | AJ: 61.7 -> 62.6 (0.9 AJ) | Helped |
Surprising findings
- Cost-volume slightly hurts in-domain Kubric performance (0.2 AJ drop) while significantly helping out-of-domain DAVIS, suggesting its benefit is mainly domain-gap mitigation.
- Sharing attention weights between content and position updates is detrimental to optimization, motivating a separate disentangler MLP.
Limitations
- Decoder self-attention is vanilla quadratic in number of queries, limiting dense point tracking.
- Evaluation uses 256x256 input resolution for fair comparison, and results are for the online First mode unless noted.
Experimental setup
Datasets: Kubric TAP-Vid-DAVIS TAP-Vid-Kinetics
Benchmarks: TAP-Vid-DAVIS TAP-Vid-DAVIS-S (strided) TAP-Vid-Kinetics Kubric in-domain evaluation set
Metrics: AJ <delta_avg^x OA
Baselines: PIPs TAP-Net Context-PIPs MFT TAPIR OmniMotion CoTracker CoTracker2 TAPTR BootsTAP
Builds on: TAPTR DETR deformable attention RAFT
Closest papers by meaning
- TAPVid-3D: A Benchmark for Tracking Any Point in 3D (NeurIPS 2024 · similarity 0.87)
- TAPIP3D: Tracking Any Point in Persistent 3D Geometry (NeurIPS 2025 · similarity 0.86)
- ETAP: Event-based Tracking of Any Point (CVPR 2025 · similarity 0.86)
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.85)
- A Distractor-Aware Memory for Visual Object Tracking with SAM2 (CVPR 2025 · similarity 0.84)
- TAPVid-360: Tracking Any Point in 360 from Narrow Field of View Video (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.