Research map / Detection and segmentation / Tracking
Sequential Fusion Based Multi-Granularity Consistency for Space-Time Transformer Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STCFormer, a space-time transformer tracker that trains with a sequential fusion cycle over template and search frames plus label-, attention-, and feature-level consistency losses (LCL, ACL, SCL) built on OSTrack.
Problem
Visual object tracking mostly exploits spatial template matching and under-uses temporal context; existing temporal methods add computational cost or depend on template selection quality.
Approach
A cyclic sequential-fusion training circuit that updates the template from prior predicted boxes across chronological frames and closes the loop back to the first search frame, combined with three multi-granularity consistency losses (label, attention, semantic) that use ground-truth labels to separate target and background regions. The temporal modeling is training-only and adds no inference cost
Key results
- STCFormer reaches 52.0% AUC on LaSOText, outperforming the OSTrack-384 baseline (50.5%) by 1.5 points.
- On TNL2K, STCFormer achieves 57.7% AUC versus 55.9% for OSTrack-384.
- On LaSOT, STCFormer reports 71.5% AUC versus 71.1% for OSTrack-384, with 55.5 fps and 92M parameters, essentially matching baseline cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LCL (ablation) | OSTrack-384 · LaSOT | AUC: 71.1 -> 71.4 | Helped |
| add ACL (ablation) | OSTrack-384 · LaSOT | AUC: 71.1 -> 71.2 | Helped |
| add SCL (ablation) | OSTrack-384 · LaSOT | AUC: 71.1 -> 71.3 | Helped |
| add LCL (ablation) | OSTrack-384 · LaSOText | AUC: 50.5 -> 51.0 | Helped |
| add ACL (ablation) | OSTrack-384 · LaSOText | AUC: 50.5 -> 51.7 | Helped |
| add SCL (ablation) | OSTrack-384 · LaSOText | AUC: 50.5 -> 50.9 | Helped |
| add LCL (ablation) | OSTrack-384 · TNL2K | AUC: 55.9 -> 57.6 | Helped |
| add ACL (ablation) | OSTrack-384 · TNL2K | AUC: 55.9 -> 57.5 | Helped |
| add SCL (ablation) | OSTrack-384 · TNL2K | AUC: 55.9 -> 57.5 | Helped |
| add LCL, ACL, and SCL (ablation) | OSTrack-384 · LaSOT | AUC: 71.1 -> 71.5 | Helped |
| add LCL, ACL, and SCL (ablation) | OSTrack-384 · LaSOText | AUC: 50.5 -> 52.0 | Helped |
| add LCL, ACL, and SCL (ablation) | OSTrack-384 · TNL2K | AUC: 55.9 -> 57.7 | Helped |
Surprising findings
- Using the predicted box b0 instead of the ground-truth box B0 in LCL was unstable and gave minimal improvement, so the ground-truth label was used instead.
Limitations
- Training cost is higher because each training circuit runs six forward processes.
- The ACL and SCL weights are sensitive, and the paper notes that the ground-truth-box variant of LCL was needed for stability.
Experimental setup
Datasets: COCO LaSOT GOT-10k TrackingNet
Benchmarks: LaSOT LaSOText TNL2K GOT-10k UAV123
Metrics: AUC PNorm precision AO SR0.5 SR0.75 FPS MACs parameters
Baselines: OSTrack-384 SwinTrack-B MixFormer AiATrack CIA GRM STARK KeepTrack TransInMo TransT
Builds on: OSTrack STARK MAE GIoU loss
Closest papers by meaning
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.87)
- Decoupled Spatio-Temporal Consistency Learning for Self-Supervised Tracking (AAAI 2025 · similarity 0.87)
- Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better (CVPR 2025 · similarity 0.86)
- UTPTrack: Towards Simple and Unified Token Pruning for Visual Tracking (CVPR 2026 · similarity 0.85)
- Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking (AAAI 2025 · similarity 0.85)
- Autoregressive Queries for Adaptive Tracking with Spatio-Temporal Transformers (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.