Research map / Detection and segmentation / Tracking
OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OneTracker, a foundation RGB tracker pretrained on LaSOT/TrackingNet/GOT-10K plus parameter-efficient prompt-tuned adapters (CMT Prompters and TTP Transformer adapters) that unify RGB and RGB+N/M/D/T/E tracking.
Problem
Visual object tracking is fragmented into task-specific models for RGB and RGB+X (language, mask, depth, thermal, event) inputs, which need customized architectures, have limited data per task, and do not share a unified temporal matching mechanism.
Approach
Pretrain a single RGB foundation tracker on large tracking datasets, then freeze it and adapt to each RGB+X modality via lightweight prompt embeddings (CMT Prompters) and low-rank adapters inserted into the linear projections of each transformer layer (TTP Transformer), treating extra modalities as prompts.
Key results
- Foundation Tracker reaches 70.5 AUC on LaSOT, outperforming listed RGB trackers such as GRM (69.9).
- Prompt Tracker with 2.8M trainable parameters reaches 69.7 AUC... reported as 60.9 PR on LasHeR and 91.5 precision on OTB99 (RGB+N), surpassing prior RGB+N trackers.
- Prompt Tracker achieves 82.5 J&F-style gains on DAVIS17 (J&F 85.6 vs 80.2 for full finetuning) while training only 2.8M parameters versus 99.83M for full finetuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain a Foundation Tracker on RGB tracking datasets | OneTracker · LaSOT | AUC: 69.9 -> 70.5 | Helped |
| add multimodal prompts and task adapters through Prompt Tracker | RGB+N tracking · OTB99 | AUC: 68.3 -> 69.7 | Helped |
| remove CMT Prompters (ablation) | Prompt Tracker · DepthTrack | F: 56.5 -> 60.9 | Hurt |
| remove CMT Prompters (ablation) | Prompt Tracker · LasHeR | PR: 60.7 -> 67.2 | Hurt |
| remove CMT Prompters (ablation) | Prompt Tracker · DAVIS17 | J &F: 80.4 -> 82.5 | Hurt |
| remove TTP Transformer (ablation) | Prompt Tracker · DepthTrack | F: 59.2 -> 60.9 | Hurt |
| remove TTP Transformer (ablation) | Prompt Tracker · LasHeR | PR: 65.6 -> 67.2 | Hurt |
| remove TTP Transformer (ablation) | Prompt Tracker · DAVIS17 | J &F: 81.7 -> 82.5 | Hurt |
| increase the number of CMT Prompter layers (ablation) | Prompt Tracker · DepthTrack | F: 56.5 -> 60.9 | Helped |
| increase the number of CMT Prompter layers (ablation) | Prompt Tracker · DAVIS17 | J &F: 82.5 -> 48.3 | Hurt |
| separate CMT Prompters and TTP Transformer layers by modality during joint training (ablation) | Prompt Tracker · RGB+X tracking | F | Helped |
| train Prompt Tracker on the corresponding task-specific data (ablation) | Prompt Tracker · DepthTrack | F: 58.6 -> 60.9 | Helped |
Surprising findings
- Increasing the number of CMT Prompter layers improves most RGB+X tracking tasks but significantly degrades RGB+M (mask) tracking performance.
Limitations
- RGB+M tracking relies on a first-frame and previous-frame matching scheme and has a remaining gap to memory-based specialized VOS models.
- Adding more CMT Prompter layers hurts RGB+M performance, indicating mask prompts do not benefit from deep integration.
Experimental setup
Datasets: LaSOT TrackingNet GOT-10K DepthTrack VOT-RGBD2022 LasHeR RGBT234 VisEvent TNL2K OTB99 DAVIS16 DAVIS17
Benchmarks: LaSOT TrackingNet OTB99 TNL2K DepthTrack VOT-RGBD2022 LasHeR RGBT234 VisEvent DAVIS16 DAVIS17
Metrics: AUC normalized precision precision F-score EAO accuracy robustness PR SR MPR
Baselines: TransT STARK MixFormer OSTrack GRM UniTrack UNINEXT ViPT Unicorn OmniTracker
Builds on: ViT OSTrack BoxInst ViPT AiATrack
Closest papers by meaning
- Single-Model and Any-Modality for Video Object Tracking (CVPR 2024 · similarity 0.89)
- SUTrack: Towards Simple and Unified Single Object Tracking (AAAI 2025 · similarity 0.87)
- UTPTrack: Towards Simple and Unified Token Pruning for Visual Tracking (CVPR 2026 · similarity 0.86)
- Temporal Adaptive RGBT Tracking with Modality Prompt (AAAI 2024 · similarity 0.86)
- VastTrack: Vast Category Visual Object Tracking (NeurIPS 2024 · similarity 0.86)
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.