Research map / Vision-language and multimodal / Cross-modal learning
Bi-directional Adapter for Multimodal Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BAT, a multi-modal (RGB-T) visual prompt tracker that uses a universal bi-directional adapter to cross-prompt two modality-specific branches of a frozen pre-trained tracker, adding only 0.32M trainable parameters.
Problem
RGB-thermal (RGB-T) object tracking relies on scarce multi-modal data, and existing prompt-based methods fix one dominant modality even though the dominant modality changes with the environment.
Approach
A lightweight hourglass bi-directional adapter embedded in each transformer layer of two weight-shared frozen foundation-model branches passes feature prompts in both directions between RGB and TIR, so no modality is predefined as dominant.
Key results
- On RGBT234, BAT reaches 86.8% MPR and 64.1% MSR, outperforming the runner-up DMCNet (83.9% MPR) and ViPT (61.7% MSR).
- On LasHeR, BAT reaches 70.2% PR and 56.3% SR, improving over ViPT (65.1% PR, 52.5% SR) by 5.1 and 3.8 points.
- With only 0.32M trainable parameters, BAT outperforms full fine-tuning and prompt-learning baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add bi-directional adapter to frozen dual-stream foundation model (BAT vs ViPT) | multi-modal RGB-T tracker · LasHeR | PR: +5.1 over ViPT (70.2 vs 65.1) | Helped |
| add bi-directional adapter to frozen dual-stream foundation model (BAT vs ViPT) | multi-modal RGB-T tracker · RGBT234 | MPR: +3.3 over ViPT (86.8 vs 83.5) | Helped |
| dual-stream framework without adapter (Baseline-Dual) vs single foundation model (ablation) | foundation tracker · LasHeR | PR: +0.7 (52.2 vs 51.5) | No clear effect |
| add single-directional adapter (TIR->RGB or RGB->TIR) to dual-stream encoder (ablation) | foundation tracker · LasHeR | PR: +17.5 (BAT-RGB 69.0) and +17.0 (BAT-TIR 68.5) over Baseline 51.5 | Helped |
| use dual non-shared adapters instead of universal shared bi-directional adapter (ablation) | BAT · LasHeR | PR: 69.6 vs 70.2 (universal); SR 56.4 vs 56.3 | No clear effect |
| restrict adapter to only layer 1 (BAT-1) (ablation) | BAT · LasHeR | PR: 61.4 vs 70.2 (all 12 layers) | Hurt |
| place adapter in middle layers 5-8 only (BAT-4) instead of all 12 layers (ablation) | BAT · LasHeR | PR: 68.6 vs 70.2 (drop 1.6); SR 55.2 vs 56.3 | Hurt |
| place adapter in layers 1-4 only or 9-12 only (ablation) | BAT · LasHeR | PR: 65.2 (layers 1-4) and 66.4 (layers 9-12) vs 70.2 | Hurt |
| BAT vs ViPT on extreme attributes (LI, AIV) | BAT · LasHeR extreme attributes | PR/SR: LI PR 60.4 vs 49.9; AIV SR 45.3 vs 34.2 | Helped |
Surprising findings
- The dual-stream baseline without adapters is only marginally better than the single foundation model (52.2 vs 51.5 PR on LasHeR), so gains come from the adapter.
- Adapting only the middle layers (BAT-4, layers 5-8) gets 68.6 PR, close to the full 12-layer version at 70.2 PR.
Limitations
- Validated only on RGB-T tracking; generalization to more diverse modalities is untested.
- Training is run on the LasHeR training set only, and the paper does not report results with other training sets.
- Adapter-layer ablations are reported on LasHeR only.
Experimental setup
Datasets: LasHeR RGBT234
Benchmarks: RGBT234 LasHeR
Metrics: MPR MSR PR SR
Baselines: mfDiMP DAPNet CAT CMPP STARK TransT JMMAC MANet++ FANet ADRNet OSTrack-256 APFNet
Builds on: ViPT ProTrack OSTrack VPT
Closest papers by meaning
- Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm (NeurIPS 2025 · similarity 0.91)
- Single-Model and Any-Modality for Video Object Tracking (CVPR 2024 · similarity 0.88)
- SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking (CVPR 2024 · similarity 0.88)
- Cross-modulated Attention Transformer for RGBT Tracking (AAAI 2025 · similarity 0.87)
- Temporal Adaptive RGBT Tracking with Modality Prompt (AAAI 2024 · similarity 0.86)
- All-Day Multi-Camera Multi-Target Tracking (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.