Research map / Detection and segmentation / Tracking
Temporal Adaptive RGBT Tracking with Modality Prompt
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TATrack, an RGBT tracker that runs feature extraction and cross-modal interaction jointly via modality prompts in two branches (initial template and confidence-selected online template) linked by a spatio-temporal interaction self-attention.
Problem
RGBT tracking underuses temporal information: static-template trackers ignore object state changes, while online-sampling trackers suffer drift and neglect spatio-temporal correlation; cross-modal interaction is also often insufficient.
Approach
A spatio-temporal two-stream ViT where each branch (initial template vs. confidence-selected online template) performs multi-modal feature extraction and cross-modal interaction via modality-complementary prompts, with an STI self-attention over concatenated template tokens bridging the branches across frames.
Key results
- On LasHeR, TATrack reaches 70.2 precision, 66.7 normalized precision and 56.1 success, exceeding ViPT by 5.1, 5.0 and 3.6 points.
- On RGBT234, TATrack achieves 87.2 precision and 64.4 success, slightly above TBSI (87.1 / 63.7).
- On RGBT210, TATrack matches TBSI's 85.3 precision but is 0.7 lower in success (61.8 vs 62.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add modality prompts to a single-branch RGB tracker (ablation) | RGBT tracking baseline · LasHeR | Precision: 51.5 -> 65.1 | Helped |
| Add modality prompts to a single-branch RGB tracker (ablation) | RGBT tracking baseline · LasHeR | Success: 41.2 -> 52.5 | Helped |
| Update the online template each frame instead of selecting reliable updates (ablation) | Modality-prompt baseline with STI · LasHeR | Precision: 70.2 -> 66.2 | Hurt |
| Update the online template each frame instead of selecting reliable updates (ablation) | Modality-prompt baseline with STI · LasHeR | Success: 56.1 -> 52.9 | Hurt |
| Add STI to the online-template-selection model (ablation) | TATrack without STI · LasHeR | Precision: 68.5 -> 70.2 | Helped |
| Add STI to the online-template-selection model (ablation) | TATrack without STI · LasHeR | Success: 54.8 -> 56.1 | Helped |
| Add STI at layer 4 (ablation) | TATrack STI placement · LasHeR | Precision: 68.5 -> 68.6 | Helped |
| Add STI at layers 4 and 7 (ablation) | TATrack STI placement · LasHeR | Precision: 68.5 -> 69.5 | Helped |
| Add STI at layers 4, 7, and 10 (ablation) | TATrack STI placement · LasHeR | Precision: 68.5 -> 70.2 | Helped |
| Add STI at layers 4, 7, and 10 (ablation) | TATrack STI placement · LasHeR | FPS: 28.3 -> 26.1 | Hurt |
| Use TATrack rather than TBSI | RGBT tracking · LasHeR | PR: 69.2 -> 70.2 | Helped |
| Use TATrack rather than TBSI | RGBT tracking · RGBT234 | Precision: 87.1 -> 87.2 | Helped |
Surprising findings
- Updating the online template every frame degrades performance relative to confidence-based selection (ablation model 4 vs. TATrack), indicating poor-quality templates hurt tracking.
Limitations
- Online template update relies on a fixed interval (50 frames) and max classification score as the reliability criterion, which the authors note needs more options.
- Inserting STI layers reduces inference speed (28.3 to 26.1 FPS).
- Only short-term tracking is evaluated; the approach relies on a heuristic template selection without additional training.
Experimental setup
Datasets: LasHeR
Benchmarks: LasHeR RGBT234 RGBT210
Metrics: precision rate normalized precision rate success rate FPS
Baselines: HMFT mfDiMP CAT MANet MANet++ MaCNet APFNet ProTrack ViPT TBSI
Builds on: ViPT OSTrack Stark Vision Transformer
Closest papers by meaning
- CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features (ICML 2025 · similarity 0.90)
- Cross-modulated Attention Transformer for RGBT Tracking (AAAI 2025 · similarity 0.90)
- Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking (AAAI 2025 · similarity 0.89)
- RGBT Tracking via All-layer Multimodal Interactions with Progressive Fusion Mamba (AAAI 2025 · similarity 0.89)
- Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm (NeurIPS 2025 · similarity 0.88)
- Single-Model and Any-Modality for Video Object Tracking (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.