Research map / Vision-language and multimodal / Cross-modal learning
Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RGBDT500, a 500-video tri-modal (RGB, depth, thermal) tracking dataset and RDTTrack, a baseline that adds depth-TIR orthogonal-projection fusion as prompts to a frozen OSTrack.
Problem
Existing multi-modal trackers fuse only dual modalities (RGB-D or RGB-T), limiting robustness in complex scenes; no tri-modal tracking dataset or method exists.
Approach
Fuse depth and thermal features via an orthogonal projection constraint, then feed the fused D-TIR tokens together with RGB tokens as multi-modal visual prompts into a frozen pre-trained RGB tracker (OSTrack), training only the fusion and prompt modules.
Key results
- RDTTrack reaches 0.752 AUC and 0.792 DP on RGBDT500, versus 0.732 AUC / 0.781 DP for the best RGB-only tracker MixFormer and 0.732 / 0.790 for the best RGB-T tracker UnTrack_RGBT.
- RDTTrack runs at 76.6 FPS on an RTX 3090 with only 0.86M trainable parameters.
- Removing the depth-TIR orthogonal projection drops AUC from 0.752 to 0.733.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add depth and thermal input to the RGB tracker (ablation) | RDTTrack · RGBDT500 test | AUC: 0.737 -> 0.752 | Helped |
| Add thermal input to the RGB+Depth tracker (ablation) | RDTTrack · RGBDT500 test | DP: 0.776 -> 0.792 | Helped |
| Add depth input to the RGB+Thermal tracker (ablation) | RDTTrack · RGBDT500 test | AUC: 0.734 -> 0.752 | Helped |
| Add depth input to the RGB+Thermal tracker (ablation) | RDTTrack · RGBDT500 test | DP: 0.774 -> 0.792 | Helped |
| Add Depth-TIR orthogonal projection (ablation) | RDTTrack · RGBDT500 test | AUC: 0.733 -> 0.752 | Helped |
| Add Depth-TIR orthogonal projection (ablation) | RDTTrack · RGBDT500 test | DP: 0.773 -> 0.792 | Helped |
| Add learnable α and β to orthogonal projection (ablation) | RDTTrack · RGBDT500 test | AUC: 0.739 -> 0.752 | Helped |
| Add learnable α and β to orthogonal projection (ablation) | RDTTrack · RGBDT500 test | DP: 0.779 -> 0.792 | Helped |
| Use tri-modal RGB+Depth+Thermal rather than RGB-only | RDTTrack · RGBDT500 test | AUC: +2.0% | Helped |
| Use tri-modal RGB+Depth+Thermal rather than RGB-only | RDTTrack · RGBDT500 test | DP: +1.1% | Helped |
| Use tri-modal RGB+Depth+Thermal rather than RGB+Thermal | RDTTrack · RGBDT500 test | AUC: +3.9% | Helped |
| Use tri-modal RGB+Depth+Thermal rather than RGB+Thermal | RDTTrack · RGBDT500 test | DP: +1.0% | Helped |
Surprising findings
- Retrained dual-modal SPT and ViPT on RGBDT500 show only marginal gains over their RGB-only baselines (e.g., ViPT_RGBT 0.729 vs OSTrack 0.694 AUC, but SPT_RGBD 0.706 vs STARK 0.692).
Limitations
- RDTTrack cannot handle a variable number of input modalities.
- Dataset covers only RGB, depth, and TIR; other modalities such as LiDAR or event are not included.
- Training-set annotations are sparse, with only K-means-selected representative frames annotated.
Experimental setup
Datasets: RGBDT500
Benchmarks: RGBDT500
Metrics: AUC DP FPS
Baselines: OSTrack MixFormer SeqTrack ToMP ViPT SPT SDSTrack UnTrack BAT TBSI DCEvo
Builds on: OSTrack visual prompt learning focal loss GIoU loss
Closest papers by meaning
- Single-Model and Any-Modality for Video Object Tracking (CVPR 2024 · similarity 0.92)
- Bi-directional Adapter for Multimodal Tracking (AAAI 2024 · similarity 0.91)
- Temporal Adaptive RGBT Tracking with Modality Prompt (AAAI 2024 · similarity 0.88)
- SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking (CVPR 2024 · similarity 0.88)
- RGBT Tracking via All-layer Multimodal Interactions with Progressive Fusion Mamba (AAAI 2025 · similarity 0.88)
- Cross-modulated Attention Transformer for RGBT Tracking (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.