Research map / Detection and segmentation / Tracking
Single-Model and Any-Modality for Video Object Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Un-Track trains a single set of parameters for RGB-tracking with any auxiliary modality (depth, thermal, event) by learning a shared low-rank embedding from RGB-X pairs plus outer modal prompting and LoRA finetuning of a frozen OSTrack-style RGB tracker.
Problem
Multimodal video object tracking usually needs modality-specific parameter sets; unifying RGB with depth, thermal and event inputs is hard due to heterogeneous representations, scarce multimodal datasets, and missing modalities.
Approach
Learn a shared embedding for any auxiliary modality via low-rank factorization/reconstruction guided by explicit edge (gradient) maps, trained only on RGB-X pairs, combined with a shrinkage-based token prompting block and LoRA finetuning of a frozen pretrained RGB tracker, so one parameter set handles any or missing modality.
Key results
- On DepthTrack, Un-Track reaches F-score 0.610 in the single-parameter-set setting, versus 0.594 for ViPT (uni-model) and 0.612 for ViPT depth-specific, at +6.65M parameters and +2.14 GFLOPs over the RGB baseline.
- On LasHeR, the uni-model achieves a +3.8% absolute precision gain over ViPT in the cross-domain single-parameter setting.
- On VisEvent, the uni-model achieves +1.1% absolute precision over the current SOTA uni-model variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a single Un-Track model trained across modalities instead of the ViPT uni-model | Un-Track on DepthTrack · DepthTrack test set | F-score: 0.561 -> 0.610 | Helped |
| use a single Un-Track model trained across modalities instead of the ViPT uni-model | Un-Track on Lasher · Lasher thermal testing set | PR: 0.608 -> 0.646 | Helped |
| use a single Un-Track model trained across modalities instead of the ViPT uni-model | Un-Track on VisEvent · VisEvent | Precision: +1.1% absolute | Helped |
| use the mixed-modality Un-Track model instead of depth-specific ViPT | Un-Track on VOT-RGBD2022 · VOT-RGBD2022 | Accuracy: +0.5% absolute | Helped |
| use the mixed-modality Un-Track model instead of thermal-specific ViPT | Un-Track on RGBT234 · RGBT234 | MPR: 0.835 -> 0.842 | Helped |
| add the shared embedding rather than feeding mixed modalities directly (ablation) | Un-Track on DepthTrack · DepthTrack test set | F-score: 0.599 -> 0.610 | Helped |
| add the proposed modal-prompting block instead of the [91] counterpart (ablation) | Un-Track on DepthTrack · DepthTrack test set | F-score: 0.579 -> 0.610 | Helped |
| add LoRA finetuning (ablation) | Un-Track on DepthTrack · DepthTrack test set | F-score: 0.594 -> 0.610 | Helped |
| add explicit edge guidance to the shared embedding (ablation) | Un-Track on DepthTrack · DepthTrack test set | F-score: 0.600 -> 0.610 | Helped |
| add implicit learning modules to the shared embedding (ablation) | Un-Track on DepthTrack · DepthTrack test set | F-score: 0.604 -> 0.610 | Helped |
| retain in-domain approximation and fusion rather than bypassing them (ablation) | Un-Track on DepthTrack · DepthTrack test set | F-score: 0.581 -> 0.610 | Helped |
Surprising findings
- Naively training ViPT jointly on all modalities with a single parameter set significantly degrades its performance on DepthTrack.
- UniNext, a leading large-scale tracker, performs poorly on RGB-D downstream settings despite strong general tracking results.
Limitations
- Trained on a limited set of sparse downstream multimodal datasets (DepthTrack, LasHeR, VisEvent) without a dataset covering all modality combinations.
- Gains on the depth-specific setting are small relative to the uni-model baseline and depend on hyperparameters such as rank and percentile chosen by ablation.
Experimental setup
Datasets: DepthTrack LasHeR VisEvent
Benchmarks: DepthTrack LasHeR VisEvent VOT-RGBD2022 RGBT234
Metrics: F-score precision recall success rate EAO accuracy robustness MPR MSR
Baselines: ViPT ProTrack UniNext SeqTrack OSTrack AiATrack Stark
Builds on: OSTrack ViPT LoRA
Closest papers by meaning
- Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm (NeurIPS 2025 · similarity 0.92)
- SUTrack: Towards Simple and Unified Single Object Tracking (AAAI 2025 · similarity 0.89)
- OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning (CVPR 2024 · similarity 0.89)
- Bi-directional Adapter for Multimodal Tracking (AAAI 2024 · similarity 0.88)
- SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking (CVPR 2024 · similarity 0.88)
- CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.