Research map / Vision-language and multimodal / Cross-modal learning
RGBT Tracking via All-layer Multimodal Interactions with Progressive Fusion Mamba
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AINet, a ViT-based RGBT tracker that uses Difference-based Fusion Mamba (DFM) at every layer and an Order-dynamic Fusion Mamba (OFM) to interact tokens from all layers, reaching SOTA on four RGBT benchmarks.
Problem
Existing RGBT trackers can only fuse modalities at a few layers because interaction modules (Transformer attention or heavy adapters) are too costly to apply across all layers, limiting use of all-layer multimodal information.
Approach
Use linear-complexity Mamba to model RGB-TIR feature differences at every backbone layer (DFM) and to scan the concatenated tokens of all layers with an input-dependent, dynamically predicted layer order (OFM) so early-layer information is not forgotten.
Key results
- AINet (384x384) reaches 74.2 PR / 70.1 NPR / 59.1 SR on LasHeR, versus 70.7 PR / 56.6 SR for GMMT, the strongest prior tracker reported there.
- On VTUAV, AINet (384x384) achieves 88.0 PR / 75.3 SR, 7.9/8.6 points above QAT.
- On LasHeR, AINet uses 15.9M params and 5.2G FLOPs at 38.1 FPS, versus GMMT's 962.2M params and 146.5G FLOPs at 22.4 FPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AINet (384×384) instead of QAT | RGBT tracking · RGBT210 | PR: 87.5 vs 86.8 | Helped |
| use AINet (384×384) instead of QAT | RGBT tracking · RGBT234 | PR: 89.2 vs 88.4 | Helped |
| use AINet (384×384) instead of GMMT | RGBT tracking · LasHeR | PR: 74.2 vs 70.7 | Helped |
| use AINet (384×384) instead of QAT | RGBT tracking · VTUAV | PR: 88.0 vs 80.1 | Helped |
| add DFM to the baseline (ablation) | AINet · LasHeR | PR: 71.1 -> 72.1 | Helped |
| add OFM to the baseline with simple feature addition (ablation) | AINet · LasHeR | PR: 71.1 -> 72.2 | Helped |
| add both DFM and OFM to the baseline (ablation) | AINet · LasHeR | PR: 71.1 -> 73.0 | Helped |
| add order-dynamic scanning to DFM and OFM (ablation) | AINet · LasHeR | PR: 72.5 -> 73.0 | Helped |
| increase input resolution from 256×256 to 384×384 (ablation) | AINet · LasHeR | PR: 73.0 -> 74.2 | Helped |
| use all 12 layers instead of only layer 11 for fusion (ablation) | AINet · LasHeR | PR: 71.4 -> 73.0 | Helped |
Surprising findings
- The Transformer-based variant's GPU memory grows quadratically with resolution and runs out of memory at five layers, while AINet scales linearly.
- Despite lower FLOPs, AINet runs slower than SDSTrack (38.1 vs 18.4 FPS is reported as higher, but the authors note Mamba is not yet well accelerated on available hardware for speed relative to their expectation).
Limitations
- Backbone is still ViT, so overall model efficiency could be improved; authors plan a pure Mamba backbone.
- Mamba is not yet adapted to available acceleration hardware, so inference speed lags behind FLOPs advantage.
- No pure Mamba tracking pretrained model exists, requiring future distillation.
Experimental setup
Datasets: LasHeR training set VTUAV training set
Benchmarks: RGBT210 RGBT234 LasHeR VTUAV
Metrics: PR NPR SR FPS Params FLOPs
Baselines: TBSI GMMT QAT TATrack SDSTrack OneTracker Un-Track CMD ViPT BAT
Builds on: OSTrack Mamba DropTrack
Closest papers by meaning
- Exploring Historical Information for RGBE Visual Tracking with Mamba (CVPR 2025 · similarity 0.89)
- Cross-modulated Attention Transformer for RGBT Tracking (AAAI 2025 · similarity 0.89)
- Temporal Adaptive RGBT Tracking with Modality Prompt (AAAI 2024 · similarity 0.89)
- Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm (NeurIPS 2025 · similarity 0.88)
- Single-Model and Any-Modality for Video Object Tracking (CVPR 2024 · similarity 0.87)
- SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.