Research map / Detection and segmentation / Tracking
Track-On: Transformer-based Online Point Tracking with Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Track-On is a causal, frame-by-frame transformer for long-term point tracking that uses spatial and context memory modules and patch classification plus offset refinement, reaching state of the art among online trackers and competitive with offline ones.
Problem
Long-term point tracking across video frames under occlusion, appearance change, and lighting/perspective shifts, in an online streaming setting where future frames are unavailable and offline methods are memory-heavy.
Approach
Treat tracked points as transformer decoder queries, replace iterative regression with patch classification (similarity over patches, with re-ranking of top-k) plus local offset refinement, and propagate temporal information through two FIFO memories: spatial memory that refreshes the initial query from recent predicted-location features, and context memory that stores past decoded queries for occ
Key results
- On TAP-Vid DAVIS, Track-On reaches 65.0 AJ and 78.0 delta_avg, beating all online models (Online TAPIR 56.7 AJ) and offline models such as CoTracker3 (64.5 AJ).
- On TAP-Vid RGB-Stacking, Track-On achieves 71.4 AJ, 85.2 delta_avg and 91.7 OA, exceeding Online TAPIR by 3.7 AJ and CoTracker3 (window) on delta_avg and OA.
- Ablation: adding spatial and context memory raises AJ from 52.0 (no memory) to 64.9 on DAVIS, and inference-time memory extension adds a further small gain to 65.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Track-On instead of CoTracker3 (Window) | TAP-Vid DAVIS · TAP-Vid DAVIS | AJ: 64.5 -> 65.0 | Helped |
| use Track-On instead of CoTracker3 (Video) | TAP-Vid RGB-Stacking · TAP-Vid RGB-Stacking | AJ: 74.0 -> 71.4 | Hurt |
| use Track-On instead of CoTracker3 (Window, Kub-L + R) | TAP-Vid Kinetics · TAP-Vid Kinetics | AJ: 55.8 -> 53.9 | Hurt |
| remove the re-ranking module (ablation) | Track-On · TAP-Vid DAVIS, without IME | AJ: 64.9 -> 62.8 | Hurt |
| add the offset head (ablation) | Track-On · TAP-Vid DAVIS, without IME | δ^1px: 27.6 -> 45.5 | Helped |
| replace deformable attention in the visibility head with a 2-layer MLP (ablation) | Track-On · TAP-Vid DAVIS, without IME | OA: 90.6 -> 90.6 | No clear effect |
| replace deformable attention in the visibility head with a 2-layer MLP (ablation) | Track-On · TAP-Vid DAVIS, without IME | AJ: 64.9 -> 64.0 | Hurt |
| add spatial memory (ablation) | Track-On without memory · TAP-Vid DAVIS, without IME | AJ: 52.0 -> 63.5 | Helped |
| add context memory (ablation) | Track-On without memory · TAP-Vid DAVIS, without IME | AJ: 52.0 -> 64.3 | Helped |
| add both spatial and context memory (ablation) | Track-On without memory · TAP-Vid DAVIS, without IME | AJ: 52.0 -> 64.9 | Helped |
| extend memory at inference time (ablation) | Track-On with spatial and context memory · TAP-Vid DAVIS | AJ: 64.9 -> 65.0 | Helped |
| extend memory size to 48 (ablation) | Track-On · TAP-Vid DAVIS | AJ: 64.88 AJ → 65.01 AJ | Helped |
Surprising findings
- Context memory alone (Model C, AJ 64.3) nearly matches the full spatial plus context model, while spatial memory mainly helps localization.
- Removing the offset head drops the fraction of points tracked within 1 pixel from 45.5 to 27.6, while 16-pixel accuracy barely changes.
Limitations
- Precision loss on thin surfaces due to stride-4 feature resolution.
- Struggles to distinguish instances with similar appearance.
- Trained only on synthetic Kubric data; real-world fine-tuning (as used by some baselines) is not explored in the main results.
Experimental setup
Datasets: TAP-Vid Kubric
Benchmarks: TAP-Vid DAVIS TAP-Vid RGB-Stacking TAP-Vid Kinetics RoboTAP Dynamic Replica BADJA PointOdyssey
Metrics: AJ delta_avg (x) OA delta_vis MTE survival rate
Baselines: TAPIR Online TAPIR CoTracker3 LocoTrack TAPTR TAPTRv2 BootsTAPIR SpatialTracker MFT DynOMO
Builds on: TAP-Vid DINOv2 ViT-Adapter DETR CoTracker
Closest papers by meaning
- Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better (CVPR 2025 · similarity 0.92)
- TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation (NeurIPS 2024 · similarity 0.90)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.89)
- Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation (CVPR 2025 · similarity 0.89)
- TGFormer: Transformer with Track Query Group for Multi-Object Tracking (AAAI 2025 · similarity 0.89)
- Is This Tracker On? A Benchmark Protocol for Dynamic Tracking (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.