Research map / Vision-language and multimodal / Cross-modal learning
SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SDSTrack, a symmetric adapter-based multimodal (RGB+X) visual object tracker with complementary masked patch self-distillation, built on OSTrack and evaluated on RGB-D, RGB-T and RGB-E tracking benchmarks.
Problem
Multimodal visual object tracking fine-tunes RGB trackers either fully (overfitting, many parameters) or asymmetrically with prompt tuning (RGB dominance, poor robustness when RGB is lost); the paper targets parameter-efficient, balanced multimodal tracking that remains robust in extreme conditions.
Approach
Freeze the pre-trained OSTrack ViT and insert lightweight adapters (cross-modal adaptation for the X stream and multimodal fusion adaptation reusing late ViT blocks with a cross-modal attention mask) in a symmetric RGB-X architecture, plus a random complementary patch mask with self-distillation between clean and masked paths.
Key results
- SDSTrack reaches 0.619 precision, 0.609 recall and 0.614 F-score on DepthTrack, surpassing the previous best by 2.7%, 1.3% and 2.0%.
- On RGBT234 it achieves 84.8% MPR and 62.5% MSR, new state of the art.
- Runs at about 20.86 fps on an RTX 3090Ti while using only 14.79M tuned parameters (CMA+MFA), versus 93.60M for full fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SMA (ablation) | SDSTrack · DepthTrack | F-score: 0.554 -> 0.573 | Helped |
| add RCPM to SMA (ablation) | SDSTrack · DepthTrack | F-score: 0.573 -> 0.594 | Helped |
| add self-distillation to SMA and RCPM (ablation) | SDSTrack · DepthTrack | F-score: 0.594 -> 0.614 | Helped |
| add SMA (ablation) | SDSTrack · RGBT234 | precision: 0.799 -> 0.847 | Helped |
| add RCPM to SMA (ablation) | SDSTrack · RGBT234 | precision: 0.847 -> 0.842 | Hurt |
| add self-distillation to SMA and RCPM (ablation) | SDSTrack · RGBT234 | precision: 0.842 -> 0.848 | Helped |
| add SMA (ablation) | SDSTrack · VisEvent | precision: 0.728 -> 0.760 | Helped |
| add RCPM to SMA (ablation) | SDSTrack · VisEvent | precision: 0.760 -> 0.765 | Helped |
| add self-distillation to SMA and RCPM (ablation) | SDSTrack · VisEvent | precision: 0.765 -> 0.767 | Helped |
| use CMA rather than freezing the model (ablation) | SDSTrack · DepthTrack | F-score: 0.523 -> 0.563 | Helped |
| use MFA rather than freezing the model (ablation) | SDSTrack · DepthTrack | F-score: 0.523 -> 0.608 | Helped |
| use CMA and MFA rather than freezing the model (ablation) | SDSTrack · DepthTrack | F-score: 0.523 -> 0.614 | Helped |
Surprising findings
- Full fine-tuning on multimodal tracking data performs worse than frozen-backbone adapter tuning in the ablation (F-score 0.558 vs 0.614 on DepthTrack for adapters), suggesting overfitting under scarce data.
- Using RCPM alone gives only limited robustness gains; the gain comes mainly when RCPM is combined with self-distillation.
Limitations
- Training incurs extra computation from the self-distillation branch with masked paths.
- Inference speed (20.86 fps) is lower than ViPT (24.78 fps) in the authors' comparison.
Experimental setup
Datasets: DepthTrack LasHeR VisEvent
Benchmarks: DepthTrack VOT-RGBD2022 RGBT234 LasHeR VisEvent
Metrics: precision recall F-score EAO accuracy robustness MPR MSR success
Baselines: ViPT ProTrack SPT DeT DMTracker SBT RGBD CAT APFNet CMPP DAFNet
Builds on: OSTrack adapter tuning ViT ViPT
Closest papers by meaning
- Single-Model and Any-Modality for Video Object Tracking (CVPR 2024 · similarity 0.88)
- Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm (NeurIPS 2025 · similarity 0.88)
- Bi-directional Adapter for Multimodal Tracking (AAAI 2024 · similarity 0.88)
- Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking (AAAI 2025 · similarity 0.87)
- RGBT Tracking via All-layer Multimodal Interactions with Progressive Fusion Mamba (AAAI 2025 · similarity 0.86)
- Temporal Adaptive RGBT Tracking with Modality Prompt (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.