Research map / Architectures and efficiency / Mixture of experts
SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPMTrack, a one-stream visual tracker built on TMoE, a lightweight mixture-of-experts module applied to attention and FFN linear layers that doubles as parameter-efficient fine-tuning, extended to multi-frame spatio-temporal context.
Problem
One-stream ViT trackers use a single vanilla attention for all template-search patch relations, which cannot adaptively handle diverse relation types (background suppression vs. boundary emphasis), and full fine-tuning of large backbones is costly.
Approach
TMoE replaces linear layers in both self-attention (QKV and output projection) and FFN with mixture-of-experts consisting of a frozen shared expert, a shared trainable compression expert, and routed low-rank-style experts combined by a router; it is used as parameter-efficient fine-tuning and extended with a learnable target state token that propagates spatio-temporal context.
Key results
- SPMTrack-B reaches 74.9 AUC on LaSOT, outperforming ViT-B trackers and LoRAT-B378 (72.9) with 29.2M trainable of 115.3M parameters.
- SPMTrack-G achieves 77.4 AUC on LaSOT and 64.7 AUC on TNL2K, with SPMTrack-B reporting 86.1 AUC on TrackingNet.
- Ablation on LaSOT: TMoE over LoRA adds +0.8 AUC without spatio-temporal modeling and +1.0 AUC with it; spatio-temporal context adds +1.2 AUC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TMoE instead of using LoRA for parameter-efficient fine-tuning (ablation) | SPMTrack-B · LaSOT test | AUC (%): 72.9 -> 73.7 | Helped |
| add spatio-temporal context modeling and the target state token (ablation) | SPMTrack-B with TMoE · LaSOT test | AUC (%): 73.7 -> 74.9 | Helped |
| apply TMoE to FFN layers (ablation) | SPMTrack-B with spatio-temporal modeling and LoRA elsewhere · LaSOT test | AUC (%): 73.9 -> 74.1 | Helped |
| apply TMoE to QKV projections (ablation) | SPMTrack-B with spatio-temporal modeling and LoRA elsewhere · LaSOT test | AUC (%): 73.9 -> 74.5 | Helped |
| add TMoE to the attention output projection (ablation) | SPMTrack-B with TMoE in QKV projections · LaSOT test | AUC (%): 74.5 -> 74.6 | Helped |
| share the compression expert across routed experts (ablation) | SPMTrack-B TMoE · LaSOT test | AUC (%): 74.8 -> 74.9 | Helped |
| replace conventional MoE with TMoE (ablation) | SPMTrack-B · LaSOT test | AUC (%): 73.4 -> 74.9 | Helped |
| increase routed experts from 2 to 4 (ablation) | SPMTrack-B TMoE · GOT-10K test | AO (%): 75.3 -> 76.5 | Helped |
| increase routed experts from 4 to 6 (ablation) | SPMTrack-B TMoE · GOT-10K test | AO (%): 76.5 -> 76.6 | Helped |
| increase routed experts from 6 to 8 (ablation) | SPMTrack-B TMoE · GOT-10K test | AO (%): 76.6 -> 77.1 | Helped |
| use the same number of reference frames at training and inference (2 rather than 3) (ablation) | SPMTrack-B · GOT-10K test | AO (%): 73.1 -> 75.8 | Helped |
| use the same number of reference frames at training and inference (3 rather than 2) (ablation) | SPMTrack-B · GOT-10K test | AO (%): 74.7 -> 76.5 | Helped |
Surprising findings
- Conventional MoE in FFN layers with 316.8M params underperforms TMoE (73.4 vs 74.9 AUC on LaSOT) despite far more parameters.
- Sharing the compression expert reduces parameters while slightly improving AUC over unshared (74.9 vs 74.8).
Limitations
- Weaker performance on NfS because the method does not use consecutive target position history as in ARTrack and HIPTrack.
- Optimal separate TMoE configurations for attention and FFN layers left unexplored.
- Scaling of spatio-temporal reference frames during inference must match training frame count for best results.
Experimental setup
Datasets: LaSOT TrackingNet GOT-10K COCO
Benchmarks: LaSOT GOT-10K TrackingNet TNL2K UAV123 NfS OTB2015
Metrics: AUC PNorm Precision AO SR0.5 SR0.75
Baselines: LoRAT ARTrackV2 HIPTrack ODTrack SeqTrack ROMTrack GRM OSTrack
Builds on: DINOv2 LoRA OSTrack LoRAT Mixture of Experts
Closest papers by meaning
- ODTrack: Online Dense Temporal Token Learning for Visual Tracking (AAAI 2024 · similarity 0.88)
- MIMTrack: In-Context Tracking via Masked Image Modeling (AAAI 2025 · similarity 0.87)
- Explicit Visual Prompts for Visual Object Tracking (AAAI 2024 · similarity 0.86)
- Less Is More: Token Context-Aware Learning for Object Tracking (AAAI 2025 · similarity 0.85)
- TGFormer: Transformer with Track Query Group for Multi-Object Tracking (AAAI 2025 · similarity 0.85)
- VastTrack: Vast Category Visual Object Tracking (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.