Research map / Video / Video segmentation and tracking
CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CSTrack, a single-branch RGB-X tracker that fuses RGB and auxiliary modality tokens into a compact spatial feature via learnable queries and cross-attention, and adds a parameter-free target-heatmap-based temporal compact memory for dense temporal modeling, reaching SOTA on RGB-D/T/E benchmarks.
Problem
RGB-X tracking methods process RGB and auxiliary modalities (depth, thermal, event) in two parallel dispersed feature spaces, which complicates the architecture, makes intra-modality modeling costly, and limits inter-modality and dense temporal modeling.
Approach
Compress the RGB and X token streams into one compact spatial feature using a few modality-specific learnable queries and bidirectional cross-attention, process it with a one-stream backbone, and build a temporal memory from top-scoring search tokens selected by a parameter-free target distribution heatmap combining correlation-based and Gaussian-box cues.
Key results
- On DepthTrack CSTrack reaches 65.8 F-score vs 61.9 for the prior best SDSTrack; on VOT-RGBD22 EAO is 77.4 vs 72.8.
- On RGBT234 CSTrack achieves 70.9 MSR and 94.0 MPR, versus 65.7 and 88.3 for IPL.
- On VisEvent CSTrack reaches 65.2 SR and 82.4 PR, versus 60.8 SR and 76.7 PR for OneTracker; the compact spatial framework runs at 35 FPS with 73M params versus 136M for the symmetric dual-branch baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use compact single-branch spatial modeling instead of a dual-branch framework | CSTrack · RGBT234 | MSR: 68.2 -> 69.6 | Helped |
| remove modality-specific queries (ablation) | CSTrack SCM · DepthTrack | PR: 64.9 -> 63.7 | Hurt |
| replace cross-attention with per-modality self-attention (ablation) | CSTrack SCM · DepthTrack | PR: 64.9 -> 60.2 | Hurt |
| use separate patch embeddings instead of a shared embedding (ablation) | CSTrack SCM · DepthTrack | PR: 64.9 -> 60.9 | Hurt |
| use RoI-based temporal features (ablation) | CSTrack TCM · RGBT234 | MSR: 69.6 -> 69.0 | Hurt |
| use query-based temporal features (ablation) | CSTrack TCM · RGBT234 | MSR: 69.6 -> 69.4 | Hurt |
| select temporal tokens using h_i^t only (ablation) | CSTrack TCM · RGBT234 | MSR: 69.6 -> 70.1 | Helped |
| select temporal tokens using h_f^t only (ablation) | CSTrack TCM · RGBT234 | MSR: 69.6 -> 70.2 | Helped |
| remove RGB-only training datasets (ablation) | CSTrack training · DepthTrack | PR: 64.9 -> 60.6 | Hurt |
| train separate trackers rather than joint training (ablation) | CSTrack training · DepthTrack | PR: 64.9 -> 63.8 | Hurt |
Surprising findings
- Generic RoI-based and query-based temporal compression both degraded performance relative to the baseline without temporal memory, while heatmap-based token selection improved it.
- A classifier on the learnable query tokens identifies the input modality type (RGB-only, RGB-D, RGB-T, RGB-E) with 98-100% accuracy.
Limitations
- Does not address visual-language (RGB-L) tracking with text auxiliary modality
- Modality-missing handling simply clones the available modality rather than using a dedicated module
Experimental setup
Datasets: DepthTrack LasHeR VisEvent LaSOT GOT-10K COCO TrackingNet VastTrack TNL2k
Benchmarks: DepthTrack VOT-RGBD2022 LasHeR RGBT234 VisEvent LasHeR-Miss RGBT234-Miss
Metrics: F-score recall precision rate EAO accuracy robustness success rate mean success rate mean precision rate params
Baselines: ViPT UnTrack OneTracker SDSTrack BAT TBSI IPL ProTrack APFNet CAT
Builds on: HiViT Fast-iTPN TATrack SDSTrack ViPT
Closest papers by meaning
- Temporal Adaptive RGBT Tracking with Modality Prompt (AAAI 2024 · similarity 0.90)
- Single-Model and Any-Modality for Video Object Tracking (CVPR 2024 · similarity 0.88)
- Cross-modulated Attention Transformer for RGBT Tracking (AAAI 2025 · similarity 0.87)
- Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking (AAAI 2025 · similarity 0.86)
- RGBT Tracking via All-layer Multimodal Interactions with Progressive Fusion Mamba (AAAI 2025 · similarity 0.85)
- Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.