Research map / Detection and segmentation / Tracking
Autoregressive Sequential Pretraining for Visual Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARP, an unsupervised video pretraining that autoregressively generates future-frame target appearance with a diffusion model and back-tracks trajectories, then fine-tunes it inside ARTrackV2 as ARPTrack.
Problem
Image-level pretrained backbones used in sequential-generation visual object trackers do not capture the temporal dynamics of consecutive video frames that tracking depends on.
Approach
Pretrain a ViT encoder on video clips with two autoregressive objectives: a conditional diffusion model that predicts the next-frame target appearance from historical appearance embeddings, and a reverse-time "backtracking" trajectory verification that regenerates initial coordinates using predicted appearances as templates.
Key results
- ARPTrack-L384 reaches 81.5% AO on GOT-10k, surpassing ARTrackV2-L384 (79.5%) and becoming the first tracker above 80% AO.
- ARPTrack256 achieves 77.7% AO on GOT-10k versus 75.9% for ARTrackV2256 with the same architecture and FLOPs (51G, 91 FPS).
- Scaling pretraining data from VastTrack (76.8 AO) to K700 (77.7 AO) yields consistent gains; ARP outperforms MAE-pretrained ARTrackV2 (75.9 AO) in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain the tracker with ARP instead of MAE | OSTrack256 · GOT-10k test | AO: 71.0 -> 72.5 | Helped |
| Pretrain the tracker with ARP instead of MAE | ARTrack256 · GOT-10k test | AO: 73.5 -> 75.2 | Helped |
| Pretrain the tracker with ARP instead of MAE | ARTrackV2-256 · GOT-10k test | AO: 75.9 -> 76.8 | Helped |
| Add trajectory verification to appearance prediction (ablation) | ARP · GOT-10k test | AO: 76.3 -> 76.8 | Helped |
| Increase fixed sequence length from 3 to 7 (ablation) | ARP · GOT-10k test | AO: 74.6 -> 76.5 | Helped |
| Increase sequence length from 7 to 9 (ablation) | ARP with fixed-length sequences · GOT-10k test | AO: 76.5 -> 76.4 | Hurt |
| Use progressive instead of fixed sequence length at length 3 (ablation) | ARP · GOT-10k test | AO: 74.6 -> 73.7 | Hurt |
| Use progressive instead of fixed sequence length at length 7 (ablation) | ARP · GOT-10k test | AO: 76.5 -> 76.8 | Helped |
| Generate future-frame appearance rather than current-frame appearance (ablation) | ARP with diffusion generation · GOT-10k test | AO: 76.1 -> 76.8 | Helped |
| Use diffusion rather than masking to generate future-frame appearance (ablation) | ARP · GOT-10k test | AO: 76.4 -> 76.8 | Helped |
| Add center jitter (ablation) | ARP pretraining · GOT-10k test | AO: 73.9 -> 76.2 | Helped |
| Add scale jitter after center jitter (ablation) | ARP pretraining · GOT-10k test | AO: 76.2 -> 76.8 | Helped |
Surprising findings
- Predicting future-frame appearance outperforms predicting current-frame appearance (76.8 vs 76.1 AO), and diffusion beats MAE-style masked reconstruction in the same setup.
- Performance decreases at sequence length 9 compared with 7, suggesting longer clips introduce interference.
Limitations
- Evaluated only on tracking tasks, which rely on coordinate-based bounding box representations.
- Pretraining cost is high (29.3 days on 8 A6000 GPUs for K700).
- Performance drops at longer sequence lengths (9), indicating sensitivity to clip length.
Experimental setup
Datasets: VastTrack Kinetics-400 Kinetics-600 Kinetics-700 GOT-10k TrackingNet LaSOT
Benchmarks: GOT-10k LaSOT LaSOText TrackingNet
Metrics: AO SR0.5 SR0.75 AUC precision normalized precision FPS FLOPs memory usage
Baselines: ARTrackV2 SeqTrack DropMAE MAE OSTrack MixViT AQATrack DiffusionTrack STARK SwinTrack
Builds on: ARTrackV2 ARTrack MAE DropMAE OSTrack
Closest papers by meaning
- Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision (NeurIPS 2025 · similarity 0.86)
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.86)
- Explicit Visual Prompts for Visual Object Tracking (AAAI 2024 · similarity 0.86)
- Generative Pre-trained Autoregressive Diffusion Transformer (NeurIPS 2025 · similarity 0.86)
- Autoregressive Queries for Adaptive Tracking with Spatio-Temporal Transformers (CVPR 2024 · similarity 0.86)
- E-Motion: Future Motion Simulation via Event Sequence Diffusion (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.