Research map / Detection and segmentation / Tracking
Less Is More: Token Context-Aware Learning for Object Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LMTrack, an autoregressive visual tracker that keeps only the most important reference tokens via a Token Context Memory module and uses unidirectional attention from reference tokens to search tokens.
Problem
Frame-level context in visual object tracking treats whole reference frames as units, so redundant and noisy background patches degrade tracking accuracy and increase computation.
Approach
Scores every reference token by its importance (cross-attention multiplied by classification map, averaged over layers and heads), keeps the top-k tokens, merges in the current search tokens, and updates the reference set autoregressively; unidirectional attention lets only reference tokens influence search tokens.
Key results
- LMTrack_384 reaches 80.1% AO on GOT-10K versus 77.5% for ARTrackV2 (previous best).
- LMTrack_384 achieves 73.2% AUC on LaSOT and 85.7% AUC on TrackingNet.
- Unidirectional attention improves AO from 73.0 to 73.9 over bidirectional attention in the ablation, and autoregressive TCM tracking reaches 76.3 AO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LMTrack₃₈₄ instead of ARTrackV2₃₈₄ | Object tracking · GOT-10K test | AO: 77.5 -> 80.1 | Helped |
| Use LMTrack₃₈₄ instead of ODTrack₃₈₄ | Object tracking · LaSOT | AUC: 73.2 -> 73.2 | No clear effect |
| Use LMTrack₃₈₄ instead of HIPTrack₃₈₄ | Object tracking · LaSOTₑₓₜ | AUC: 53.0 -> 53.6 | Helped |
| Use LMTrack₃₈₄ instead of ARTrackV2₃₈₄ | Object tracking · TrackingNet | AUC: 85.7 -> 85.7 | No clear effect |
| Use LMTrack₃₈₄ instead of ODTrack₃₈₄ | Object tracking · VOT2020 | EAO: 0.581 -> 0.586 | Helped |
| Replace bidirectional attention with unidirectional attention (ablation) | LMTrack₍₂₅₆₎ without autoregression or updates · GOT-10K test | AO: 73.0 -> 73.9; 0.9% | Helped |
| Add template update (ablation) | Unidirectional attention without autoregression · GOT-10K test | AO: 73.9 -> 74.1 | Helped |
| Replace template update with TCM (ablation) | Unidirectional attention without autoregression · GOT-10K test | AO: 74.1 -> 75.0; 0.9% | Helped |
| Add autoregressive tracking (ablation) | Unidirectional attention with template update · GOT-10K test | AO: 74.1 -> 75.6; 1.5% | Helped |
| Add autoregressive tracking (ablation) | Unidirectional attention with TCM · GOT-10K test | AO: 75.0 -> 76.3; 1.3% | Helped |
| Replace template update with TCM (ablation) | Autoregressive tracking with unidirectional attention · GOT-10K test | AO: 75.6 -> 76.3; 0.7% | Helped |
Surprising findings
- Most background reference tokens are rarely referenced during tracking, while a small set of target tokens is retained long term.
Limitations
- Reference-token collection depends on the attention and classification maps, so the accuracy of the importance estimate is critical.
- Gains over SeqTrack and ODTrack are limited on some metrics, e.g. TrackingNet P_Norm and VOT2020 accuracy.
Experimental setup
Datasets: LaSOT GOT-10k TrackingNet COCO
Benchmarks: GOT-10K LaSOT LaSOT_ext TrackingNet VOT2020
Metrics: AO SR0.5 SR0.75 AUC Normalized precision Precision EAO Accuracy Robustness
Baselines: SeqTrack ODTrack ARTrackV2 HIPTrack AQATrack OSTrack STARK Mixformer SiamRCNN DiMP
Builds on: OSTrack SeqTrack STARK ViT Alpha-Refine
Closest papers by meaning
- ODTrack: Online Dense Temporal Token Learning for Visual Tracking (AAAI 2024 · similarity 0.91)
- Exploring Enhanced Contextual Information for Video-Level Object Tracking (AAAI 2025 · similarity 0.88)
- Robust Tracking via Mamba-based Context-aware Token Learning (AAAI 2025 · similarity 0.88)
- MambaLCT: Boosting Tracking via Long-term Context State Space Model (AAAI 2025 · similarity 0.88)
- MIMTrack: In-Context Tracking via Masked Image Modeling (AAAI 2025 · similarity 0.87)
- UTPTrack: Towards Simple and Unified Token Pruning for Visual Tracking (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.