Research map / Detection and segmentation / Tracking
Autoregressive Queries for Adaptive Tracking with Spatio-Temporal Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AQATrack, a single-object tracker using learnable autoregressive target queries and a temporal decoder with temporal attention, fused with spatial features by a parameter-free STM module, achieving SOTA on six tracking benchmarks.
Problem
Visual object tracking must capture complex target appearance changes over time, but top trackers rely on hand-crafted spatio-temporal components (update strategies, memory, feature concatenation) that add hyperparameters and computation and may accumulate errors.
Approach
Use a sliding window of learnable, autoregressive target queries decoded with temporal attention over previous queries, then fuse them with spatial search features through a parameter-free dot-product STM module, avoiding hand-designed update rules.
Key results
- AQATrack-256 reaches 71.4% AUC on LaSOT and AQATrack-384 reaches 72.7%, surpassing same-resolution trackers.
- AQATrack-256 achieves 57.8% AUC on TNL2K, a new SOTA at that resolution.
- Ablation on LaSOT: removing the temporal decoder/STM drops AUC from 71.4% to 70.5%; the HiViT-based variant beats the ViT-based one by 0.8% AUC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AQATrack-384 instead of ARTrack-384 | visual tracking · LaSOT test | AUC: 72.6 -> 72.7 | Helped |
| use AQATrack-384 instead of ARTrack-384 | visual tracking · LaSOText test | AUC: 51.9 -> 52.7 | Helped |
| use AQATrack-384 instead of ARTrack-384 | visual tracking · GOT-10k test | AO: 75.5 -> 76.0 | Helped |
| use AQATrack-384 instead of ARTrack-384 | visual tracking · TNL2K test | AUC: 59.8 -> 59.3 | Hurt |
| use AQATrack-384 instead of F-BDMTrack-384 | visual tracking · UAV123 test | AUC: 70.9 -> 71.2 | Helped |
| use AQATrack-384 instead of ARTrack | visual tracking · TrackingNet test | AUC: 85.1 -> 84.8 | Hurt |
| add temporal decoder and STM (ablation) | HiViT-based tracker · LaSOT test | AUC: 70.5 -> 71.4 | Helped |
| add temporal decoder to a ViT spatial encoder (ablation) | ViT-based tracker · LaSOT test | AUC: 69.1 -> 69.6 | Helped |
| add temporal decoder to a HiViT spatial encoder (ablation) | HiViT-based tracker · LaSOT test | AUC: 70.5 -> 71.4 | Helped |
| replace ViT with HiViT as spatial encoder (ablation) | AQATrack · LaSOT test | AUC: 69.6 -> 71.4 | Helped |
| use 3 rather than 1 temporal decoder layer (ablation) | AQATrack · LaSOT test | AUC: 70.5 -> 71.4 | Helped |
| use 6 rather than 3 temporal decoder layers (ablation) | AQATrack · LaSOT test | AUC: 71.4 -> 71.1 | Hurt |
Surprising findings
- Self-attention in place of temporal attention yields only a 0.9-point AUC drop, and the design gains are modest (about 0.9 AUC from temporal decoder), suggesting limited contribution of individual components.
Limitations
- Longer spatio-temporal context was not explored due to GPU memory limits.
- Performance drops with longer windows (m=8,16), attributed to small number of videos per batch and poor generalization.
- Gains over strong baselines at the same resolution are modest on some benchmarks (e.g., TrackingNet AUC).
Experimental setup
Datasets: LaSOT COCO TrackingNet GOT-10k
Benchmarks: LaSOT LaSOText GOT-10k TNL2K UAV123 TrackingNet
Metrics: AUC Pnorm precision AO SR0.5 SR0.75
Baselines: SeqTrack-B256 ARTrack-256 OSTrack-256 ROMTrack-256 MixFormer-22k STARK TransT F-BDMTrack-256
Builds on: HiViT STARK OSTrack MOTR ARTrack
Closest papers by meaning
- Temporal Adaptive RGBT Tracking with Modality Prompt (AAAI 2024 · similarity 0.87)
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.86)
- Autoregressive Sequential Pretraining for Visual Tracking (CVPR 2025 · similarity 0.86)
- Decoupled Spatio-Temporal Consistency Learning for Self-Supervised Tracking (AAAI 2025 · similarity 0.86)
- Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking (AAAI 2025 · similarity 0.86)
- CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.