Research map / Architectures and efficiency / Attention and transformers
Cross-modulated Attention Transformer for RGBT Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAFormer, an RGBT tracker whose unified cross-modulated attention modulates each modality's search-template correlation map using the other modality's correlations, plus a collaborative token elimination strategy for speed.
Problem
Transformer RGBT trackers compute per-modality search-template correlations independently, so low-quality inputs yield ambiguous, inaccurate attention weights, and separate self- and cross-attention adds redundant computation.
Approach
Treats RGB and TIR correlation maps as mutually consistent: a Correlation Modulated Enhancement module uses both modalities' ST (search-to-template) correlation parts to modulate each other, unifying self-attention and cross-attention in one attention model; a collaborative token elimination sums per-modality token similarities to prune background tokens.
Key results
- Achieves 76.9% SR on GTOT, 63.2% SR on RGBT210, and 66.4% SR on RGBT234, reported as best among compared methods.
- Reaches 88.6% PR and 76.2% SR on VTUAV, leading MACFT by 8.5%/9.4% in PR/SR.
- Runs at 83.6 FPS on a 3080ti, with CTE raising PR/SR on LasHeR to 70.0/55.6 versus 69.3/55.1 without elimination.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add cross-modal correlation modulation (ablation) | CAFormer · RGBT234 | PR: 87.5 -> 88.3 | Helped |
| Add cross-modal correlation modulation (ablation) | CAFormer · LasHeR | PR: 68.3 -> 70.0 | Helped |
| Include SS when modulating ST (ablation) | CAFormer · RGBT234 | PR: 87.6 -> 88.3 | Helped |
| Include SS when modulating ST (ablation) | CAFormer · LasHeR | PR: 69.2 -> 70.0 | Helped |
| Apply CME to ST rather than SS or TS (ablation) | CAFormer · RGBT234 | PR: 87.8 -> 88.3; 87.7 -> 88.3 | Helped |
| Apply CME jointly to ST and SS instead of ST only (ablation) | CAFormer · RGBT234 | PR: 88.3 -> 86.4 | Hurt |
| Use CAFormer blocks in the last 3 layers rather than the last 1 layer (ablation) | CAFormer · RGBT234 | PR: 87.5 -> 88.3 | Helped |
| Use CAFormer blocks in the last 6 layers instead of the last 3 layers (ablation) | CAFormer · RGBT234 | PR: 88.3 -> 86.6 | Hurt |
| Place CAFormer blocks in layers 4, 7, and 10 instead of the last 3 layers (ablation) | CAFormer · RGBT234 | PR: 88.3 -> 88.5 | Mixed |
| Use collaborative token elimination (CTE) instead of no elimination (ablation) | CAFormer · LasHeR | PR: 69.3 -> 70.0 | Helped |
| Use candidate elimination (CE) instead of no elimination (ablation) | CAFormer · LasHeR | PR: 69.3 -> 69.5 | Helped |
| Use CTE instead of CE (ablation) | CAFormer · LasHeR | PR: 69.5 -> 70.0 | Helped |
Surprising findings
- Applying the modulation to all four correlation parts jointly (ST and SS together) reduced performance relative to modulating ST alone.
- Inserting the module into 6 layers performed worse than 3 layers, with the authors attributing this to less reliable correlation priors in early layers.
Limitations
- Gains over the strongest prior trackers (e.g., TBSI, TATrack) on some benchmarks are small (e.g., 0.3%/0.7% PR/SR on RGBT210 vs TBSI)
- Authors state the correlation fusion is not yet combined with feature fusion, which they leave to future work
Experimental setup
Datasets: LasHeR VTUAV
Benchmarks: GTOT RGBT210 RGBT234 LasHeR VTUAV
Metrics: Precision Rate (PR) Success Rate (SR) Normalized Precision Rate (NPR) FPS MACs
Baselines: TBSI ViPT MACFT TATrack OneTracker Un-Track SDSTrack HMFT mfDiMP CAT
Builds on: OSTrack ViT candidate elimination
Closest papers by meaning
- Temporal Adaptive RGBT Tracking with Modality Prompt (AAAI 2024 · similarity 0.90)
- RGBT Tracking via All-layer Multimodal Interactions with Progressive Fusion Mamba (AAAI 2025 · similarity 0.89)
- Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm (NeurIPS 2025 · similarity 0.88)
- Single-Model and Any-Modality for Video Object Tracking (CVPR 2024 · similarity 0.88)
- CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features (ICML 2025 · similarity 0.87)
- Bi-directional Adapter for Multimodal Tracking (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.