Research map / Detection and segmentation / Tracking
Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STTrack, a unified multimodal (RGB+T/D/E) video object tracker that propagates autoregressive temporal state tokens via a Mamba-based temporal state generator, with background suppression interactive (BSI) and Mamba fusion modules.
Problem
Existing multimodal (RGB-T, RGB-D, RGB-E) trackers fuse spatial features or use sparse, condition-triggered template updates, failing to exploit continuous temporal correlations of targets in multimodal videos.
Approach
A temporal state generator built on cross-Mamba produces temporal information tokens each frame that are passed to the next frame, combined with attention-based background suppression inside the ViT encoder and a Mamba-based fusion of modality branches.
Key results
- STTrack achieves 60.3% SR and 76.0% PR on LasHeR, surpassing GMMT by 4.3% SR and 5.3% PR.
- On DepthTrack, STTrack reaches 63.3% F-score; on VOT-RGBD2022 it reaches 77.6 EAO, about 4.8 points above SDSTrack.
- On VisEvent, STTrack reports 61.9% AUC and 78.6% precision; runs at about 35.5 FPS on an RTX 4090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add template updates (ablation) | STTrack baseline · LasHeR | AUC: 56.0 -> 57.1 | Helped |
| Add template updates (ablation) | STTrack baseline · DepthTrack | Precision: 58.8 -> 59.5 | Helped |
| Add template updates (ablation) | STTrack baseline · VisEvent | AUC: 60.0 -> 59.8 | Hurt |
| Add temporal information (ablation) | STTrack with template updates · LasHeR | AUC: 57.1 -> 58.9 | Helped |
| Add temporal information (ablation) | STTrack with template updates · DepthTrack | Precision: 59.5 -> 62.0 | Helped |
| Add temporal information (ablation) | STTrack with template updates · VisEvent | AUC: 59.8 -> 61.1 | Helped |
| Add Mamba Fusion (ablation) | STTrack with temporal information · LasHeR | AUC: 58.9 -> 59.2 | Helped |
| Add Mamba Fusion (ablation) | STTrack with temporal information · DepthTrack | Precision: 62.0 -> 62.1 | Helped |
| Add Mamba Fusion (ablation) | STTrack with temporal information · VisEvent | AUC: 61.1 -> 61.3 | Helped |
| Add BSI Module (ablation) | STTrack with Mamba Fusion · LasHeR | AUC: 59.2 -> 60.3 | Helped |
| Add BSI Module (ablation) | STTrack with Mamba Fusion · DepthTrack | Precision: 62.1 -> 63.2 | Helped |
| Add BSI Module (ablation) | STTrack with Mamba Fusion · VisEvent | AUC: 61.3 -> 61.9 | Helped |
Surprising findings
- Using 8 temporal information tokens performs worse than 4, attributed to redundant historical information causing interference.
Limitations
- Performance drops when the number of temporal tokens grows beyond 4, indicating sensitivity to temporal token count.
- Ablation on VisEvent with filtering ratio 15% shows a large drop (51.7), suggesting instability to the BSI filtering setting.
Experimental setup
Datasets: LasHeR VisEvent DepthTrack
Benchmarks: LasHeR RGBT234 DepthTrack VOT-RGBD2022 VisEvent
Metrics: SR PR MSR MPR F-score EAO Accuracy Robustness AUC Precision
Baselines: GMMT BAT TBSI TATrack OneTracker Un-Track SDSTrack ViPT OSTrack ProTrack
Builds on: Mamba ViT ViPT OSTrack
Closest papers by meaning
- Temporal Adaptive RGBT Tracking with Modality Prompt (AAAI 2024 · similarity 0.89)
- DreamTrack: Dreaming the Future for Multimodal Visual Object Tracking (CVPR 2025 · similarity 0.87)
- SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking (CVPR 2024 · similarity 0.87)
- Single-Model and Any-Modality for Video Object Tracking (CVPR 2024 · similarity 0.86)
- CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features (ICML 2025 · similarity 0.86)
- Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.