Research map / Vision-language and multimodal / Cross-modal learning
Generative-Based Fusion Mechanism for Multi-Modal Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GMMT, a generative-based fusion mechanism using conditional GANs or diffusion models that replaces the standard feature fusion block in RGB-T and RGB-D trackers, achieving state-of-the-art on GTOT, LasHeR and RGBD1K.
Problem
Information fusion in multi-modal (RGB-T and RGB-D) visual object tracking; existing fusion blocks are fixed at inference and depend on how well they match training data.
Approach
Replace the discriminative fusion block with a generative model (conditional GAN or diffusion model) that takes the original fused features as target, with per-modality features as conditions and a random noise factor as input, so the fusion network is trained on harder instances and produces more discriminative fused features; the GM is trained in a second stage with a tracking loss plus a genera
Key results
- On LasHeR, TBSI+GMMT reaches 70.7 PR, 67.0 NPR and 56.6 SR, versus 69.2/65.7/55.6 for the TBSI baseline.
- On RGBD1K, ViPT-D*+GMMT(V) reaches an F-score of 57.4, versus 50.6 for ViPT-D* and 56.1 for SPT.
- On GTOT, TBSI+GMMT reaches 78.5 SR and 93.6 PR, versus 75.9 and 91.5 for TBSI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add diffusion-based GMMT to the Siamese tracker | Siamese tracker · LasHeR | SR: 39.8 -> 44.9 | Helped |
| Use CGAN instead of diffusion-based GMMT (ablation) | Siamese tracker · LasHeR | SR: 42.5 -> 41.9 | Hurt |
| Replace generative loss with direct L2 loss (ablation) | Siamese GMMT · LasHeR | SR: 39.8 -> 42.5 | Helped |
| Use diffusion-based GMMT rather than CGAN-based GMMT (ablation) | Siamese tracker · GTOT | PR: 81.4 -> 85.7 | Helped |
| Add GMMT to TBSI | TBSI · LasHeR | PR: 69.2 -> 70.7 | Helped |
| Add GMMT to ViPT* | ViPT* · LasHeR | SR: 52.4 -> 53.0 | Helped |
| Add GMMT to TBSI | TBSI · RGBT234 | SR: 63.8 -> 64.7 | Helped |
| Add GMMT to TBSI | TBSI · GTOT | SR: 75.9 -> 78.5 | Helped |
| Add GMMT with UViT to retrained ViPT-D | ViPT-D* · RGBD1K | F-score: 50.6 -> 57.4 | Helped |
| Replace UNet with UViT in GMMT (ablation) | ViPT-D*+GMMT · RGBD1K | F-score: 56.2 -> 57.4 | Helped |
| Increase UViT block count from 4 to 8 (ablation) | ViPT-D*+GMMT · RGBD1K | F-score: 55.4 -> 57.4 | Helped |
| Increase UNet block count from 6 to 9 (ablation) | ViPT-D*+GMMT · RGBD1K | F-score: 55.5 -> 56.2 | Helped |
Surprising findings
- Replacing the loss with a plain L2 variant (RAW) of the same larger network gives most of the SR gain on LasHeR for the Siamese baseline (42.5 vs 39.8), while the CGAN variant lowers SR slightly versus RAW (41.9 vs 42.5).
- Gains on the ViPT* baseline are small (about 0.3-0.6 SR on LasHeR), so the method helps less when the baseline already fuses modalities thoroughly.
Limitations
- Gains on strong fusion baselines such as ViPT* and TBSI are modest (about 0.3-1.0 SR on LasHeR).
- Diffusion-based GMMT requires iterative sampling, which adds computation; the paper limits it to one step.
- The method requires a two-stage training scheme with a frozen pre-trained baseline.
Experimental setup
Datasets: LasHeR GTOT RGBD1K RGBT234
Benchmarks: GTOT LasHeR RGBT234 RGBD1K
Metrics: precision rate normalised precision rate success rate recall F-score
Baselines: ViPT TBSI SiamBAN-based Siamese baseline BD2Track APFNet DMCNet HMFT ProTrack DFAT HFRD SPT DDiMP
Builds on: DDIM conditional GAN SiamBAN ViPT TBSI
Closest papers by meaning
- Diffusion Models for Multi-Task Generative Modeling (ICLR 2024 · similarity 0.86)
- Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm (NeurIPS 2025 · similarity 0.85)
- Bi-directional Adapter for Multimodal Tracking (AAAI 2024 · similarity 0.84)
- Single-Model and Any-Modality for Video Object Tracking (CVPR 2024 · similarity 0.84)
- Is Multiple Object Tracking a Matter of Specialization? (NeurIPS 2024 · similarity 0.84)
- SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.