Research map / Vision-language and multimodal / Human and embodied modeling
Dynamic Updates for Language Adaptation in Visual-Language Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DUTrack, a one-stream vision-language tracker that dynamically updates its multi-modal reference (visual template patches via attention top-k and language descriptions via an LLM) using a displacement/scale/color update rule.
Problem
Vision-language trackers rely on static multi-modal references (initial template and language annotation), which diverge from the target's changing state during long-term tracking.
Approach
Dynamically update both visual templates (top-k search patches selected by [CLS]-to-search attention in a unified HiViT/ViT encoder) and language descriptions (generated by an LLM from search image and object category), triggered by changes in target displacement, scale, and color.
Key results
- DUTrack-384 reaches 74.1% AUC on LaSOT and 65.6% AUC on TNL2K, new state of the art on the vision-language benchmarks reported.
- DUTrack-256 reaches 76.7% AO on GOT-10K, outperforming SeqTrack by 2.0%.
- Ablation on LaSOT: adding the dynamic language update lifts AUC from 71.7% to 73.0% over the static-language setting, and the dynamic template capture raises AUC from 71.0% to 71.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add dynamic template patches by setting DTCM top-k to 3 instead of 0 (ablation) | DUTrack-256 · LaSOT | AUC: 71.0 -> 71.7 | Helped |
| Add dynamic template patches by setting DTCM top-k to 3 instead of 0 (ablation) | DUTrack-256 · GOT-10k | AO: 72.2 -> 74.6 | Helped |
| Add language information and dynamically update it rather than using only initial language annotations (ablation) | DUTrack-256 with DTCM · LaSOT | AUC: 72.4 -> 73.0 | Helped |
| Increase language update frequency from parameters #2 to #4 (ablation) | DUTrack-256 with DTCM · LaSOT | AUC: 72.5 -> 73.0 | Helped |
| Replace BLIP with BLIP-2 for language generation (ablation) | DUTrack-256 · LaSOT | AUC: +0.2% | Helped |
| Use concise rather than detailed DTLLM annotations (ablation) | DUTrack-256 · LaSOT | AUC: 72.5 -> 72.9 | Helped |
| Use DUTrack-384 instead of ODTrack-B384 | LaSOT · LaSOT | AUC: 73.2 -> 74.1 | Helped |
| Use DUTrack-384 instead of AQATrack-384 | LaSOText · LaSOText | AUC: 52.7 -> 52.5 | Hurt |
| Use DUTrack-384 instead of UVLTrack-L | TNL2K · TNL2K | AUC: 64.8 -> 65.6 | Helped |
| Use DUTrack-384 instead of MMTrack-384 | OTB99-Lang · OTB99-Lang | AUC: 70.5 -> 71.3 | Helped |
| Use DUTrack-384 instead of JointNLT | MGIT · MGIT | AUC: +10% | Helped |
| Use DUTrack-384 instead of ODTrack | GOT-10k · GOT-10k | AO: 0.8% | Helped |
Surprising findings
- Detailed LLM-generated language annotations reduced performance relative to concise ones (AUC drop of 0.5%).
Limitations
- Update thresholds are set manually and require extensive tuning.
- Generated language descriptions may not be highly precise semantically.
- Gains on LaSOT over top vision-only trackers are small, attributed to small test set fluctuations.
Experimental setup
Datasets: LaSOT GOT-10K COCO TrackingNet TNL2K DTLLM-VLT annotations
Benchmarks: LaSOT LaSOText TNL2K OTB99-Lang MGIT GOT-10K UAV123
Metrics: AUC PNorm precision AO SR0.5
Baselines: JointNLT MMTrack UVLTrack QueryNLT ODTrack AQATrack SeqTrack OSTrack
Builds on: HiViT Fast-ITPN STARK OSTrack DTLLM-VLT
Closest papers by meaning
- MambaVLT: Time-Evolving Multimodal State Space Model for Vision-Language Tracking (CVPR 2025 · similarity 0.87)
- Context-Aware Integration of Language and Visual References for Natural Language Tracking (CVPR 2024 · similarity 0.87)
- ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model (NeurIPS 2024 · similarity 0.84)
- MemVLT: Vision-Language Tracking with Adaptive Memory-based Prompts (NeurIPS 2024 · similarity 0.84)
- Unifying Visual and Vision-Language Tracking via Contrastive Learning (AAAI 2024 · similarity 0.83)
- Mono3DVLT: Monocular-Video-Based 3D Visual Language Tracking (CVPR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.