Research map / Vision-language and multimodal / Cross-modal learning
Unifying Visual and Vision-Language Tracking via Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UVLTrack, a single tracker that handles BBOX, natural-language, and NL+BBOX target references with shared parameters, using a multi-modal contrastive loss and a modality-adaptive contrastive box head.
Problem
Single object tracking models are usually specialized to one target reference modality (BBOX, NL, or NL+BBOX) and generalize poorly across the others, due to the semantic gap between vision and language.
Approach
A Transformer feature extractor that encodes vision and language separately in shallow layers and fuses them in deep layers, trained with a multi-modal contrastive loss to align modalities in a shared semantic space, plus a box head that uses distribution-based cross-attention over video contexts to build target, distractor and background prototypes for contrastive localization.
Key results
- UVLTrack-L reaches 71.3 AUC on LaSOT and 64.8 AUC on TNL2K under BBOX reference, new state of the art among the visual trackers reported.
- On TNL2K under NL+BBOX, UVLTrack-L achieves 64.9 AUC versus 56.9 for JointNLT.
- UVLTrack-B runs at 58 FPS for visual tracking, about 1.46x the speed of JointNLT (39 FPS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MMCLoss (ablation) | UVLTrack-B · TNL2K (BBOX, NL, NL+BBOX) | AUC: 1.2%, 2.2% and 2.3% AUC gains | Helped |
| replace the static anchor-free head with MABH (ablation) | UVLTrack-B · TNL2K (BBOX, NL, NL+BBOX) | AUC: 2.1%, 1.9% and 1.1% AUC gains | Helped |
| use N=6 and M=6 instead of all-shallow (N=0, M=12) or early-fusion-heavy layer splits (ablation) | UVLTrack-B feature extractor · TNL2K (BBOX, NL, NL+BBOX) | AUC: 61.3; 55.1; 62.3 -> 62.7; 55.7; 63.1 | Mixed |
| use all-layer fusion (N=11, M=1) (ablation) | UVLTrack-B feature extractor · TNL2K (NL) | AUC: 55.7 -> 46.2 | Hurt |
| use the target center as the positive sample instead of average target features (ablation) | UVLTrack-B MMCLoss · TNL2K (BBOX, NL, NL+BBOX) | AUC: 61.4 -> 61.8; 53.6 -> 54.9; 61.5 -> 62.1 | Helped |
| use top-scoring out-of-box negatives instead of random negatives (ablation) | UVLTrack-B MMCLoss · TNL2K (BBOX, NL, NL+BBOX) | AUC: 61.4 -> 62.1; 53.6 -> 55.2; 61.5 -> 62.3 | Helped |
| use 9 hard-negative samples instead of 1 (ablation) | UVLTrack-B MMCLoss · TNL2K (BBOX, NL, NL+BBOX) | AUC: 61.6 -> 62.7; 54.5 -> 55.7; 61.8 -> 63.1 | Helped |
| vary distractor threshold beta from 0.25 to 0.85 (ablation) | UVLTrack-B MABH · TNL2K (BBOX, NL, NL+BBOX) | AUC: 62.1; 62.5; 62.7; 62.6 | No clear effect |
| set beta=0, aggregating all background features into one token (ablation) | UVLTrack-B MABH · TNL2K (BBOX, NL, NL+BBOX) | AUC: 62.7 -> 61.3; 55.7 -> 54.3; 63.1 -> 61.8 | Hurt |
| change the BBOX:NL:Both training ratio from 4:1:4 to 3:1:3 or 5:1:5 (ablation) | UVLTrack-B · TNL2K (BBOX, NL, NL+BBOX) | AUC: 62.3; 62.7; 62.8 | No clear effect |
| initialize the backbone with MAE rather than BERT+MAE (ablation) | UVLTrack-B · TNL2K (BBOX, NL, NL+BBOX) | AUC: 62.7 -> 62.1; 55.7 -> 53.8; 63.1 -> 61.9 | Hurt |
Surprising findings
- Vision-language trackers such as JointNLT and VLTTT perform poorly when initialized with only a bounding box (LaSOT AUC 54.5 and 53.4), suggesting over-reliance on language semantics.
Limitations
- Relies on multiple large training sets and a BERT plus MAE initialization, so gains are not isolated from data scale.
- Visual grounding results require retraining separately on each RefCOCO variant.
- Ablations are reported only on TNL2K for the UVLTrack-B variant.
Experimental setup
Datasets: LaSOT GOT-10k COCO2017 TrackingNet TNL2K OTB99 RefCOCOg
Benchmarks: TNL2K AVisT LaSOT LaSOT_ext TrackingNet NFS UAV123 OTB99 RefCOCO RefCOCO+ RefCOCOg
Metrics: AUC precision top-1 accuracy FPS
Baselines: OSTrack-384 OSTrack-256 MixFormer-L SimTrack STARK TransT JointNLT VLTTT SNLT VLTVG
Builds on: OSTrack MAE BERT VLT JointNLT
Closest papers by meaning
- ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model (NeurIPS 2024 · similarity 0.89)
- Mono3DVLT: Monocular-Video-Based 3D Visual Language Tracking (CVPR 2025 · similarity 0.87)
- MemVLT: Vision-Language Tracking with Adaptive Memory-based Prompts (NeurIPS 2024 · similarity 0.87)
- VastTrack: Vast Category Visual Object Tracking (NeurIPS 2024 · similarity 0.86)
- MVLM: Template-Free Tracking via Vision-Language Margin Confidence and Memory-Gated Tracking (CVPR 2026 · similarity 0.86)
- Single-Model and Any-Modality for Video Object Tracking (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.