Research map / Vision-language and multimodal / Human and embodied modeling
Context-Aware Integration of Language and Visual References for Natural Language Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QueryNLT, a unified multi-modal query-based tracker that modulates language and visual template prompts with historical context and decodes the target end-to-end for natural language tracking.
Problem
Tracking by natural language specification (TNL) methods match language and visual templates separately and merge results late, causing drift when the initial language description becomes misaligned with the target's dynamic state and template contains background clutter.
Approach
Modulates the language features using motion and appearance cues from a dynamic template memory, masks background in the visual template using a Gumbel-softmax subjecthood score, and fuses both prompts into object queries that a Deformable-DETR-style decoder uses to localize the target in one step.
Key results
- On TNL2K under NL setting, achieves 53.3 AUC, second only to JointNLT (54.6) and 1.2 points above reproduced JointNLT*.
- With NL+BB on TNL2K, reaches 57.8 AUC, 56.9-to-57.8 improvement over JointNLT and best across all indicators.
- Ablation shows joint matching improves AUC by 2.5 points over separate matching (49.8 to 53.3 on TNL2K).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add language modulation (ablation) | QueryNLT · TNL2K, NL | AUC: 52.2 -> 53.3 | Helped |
| add appearance modulation (ablation) | QueryNLT · TNL2K, NL | AUC: 51.6 -> 53.3 | Helped |
| integrate multimodal matching instead of separate language-search and template-search matching (ablation) | QueryNLT · TNL2K, NL | AUC: 49.8 -> 53.3 | Helped |
| use multiple dynamic templates instead of a static template (ablation) | QueryNLT · TNL2K, NL | AUC: 51.0 -> 53.3 | Helped |
| increase the query count from 1 to 3 (ablation) | QueryNLT · TNL2K, NL | AUC: 51.6 -> 52.8 | Helped |
| increase the query count from 3 to 5 (ablation) | QueryNLT · TNL2K, NL | AUC: 52.8 -> 53.3 | Helped |
| increase the query count from 5 to 7 (ablation) | QueryNLT · TNL2K, NL | AUC: 53.3 -> 53.4 | No clear effect |
| use QueryNLT instead of JointNLT | natural language tracking · OTB-Lang, NL | AUC: 59.2 -> 61.2 | Helped |
| use QueryNLT instead of TNL2K-2 | natural language tracking · OTB-Lang, NL+BB | AUC: 68.0 -> 66.7 | Hurt |
| use QueryNLT instead of JointNLT | natural language tracking · LaSOT, NL+BB | AUC: 60.4 -> 59.9 | Hurt |
| use QueryNLT instead of JointNLT | natural language tracking · TNL2K, NL+BB | AUC: 56.9 -> 57.8 | Helped |
Surprising findings
- On RefCOCOg, the tracker is second only to VLTVG, despite not being specialized for grounding.
Limitations
- LaSOT language descriptions lack relative-position cues, so the approach is less suited to language-assisted tracking there.
- Trails JointNLT in the NL-only TNL2K setting and VLTVG on RefCOCOg grounding.
Experimental setup
Datasets: TNL2K OTB-Lang LaSOT RefCOCOg
Benchmarks: TNL2K OTB-Lang LaSOT RefCOCOg
Metrics: AUC Precision Normalized precision average IoU
Baselines: JointNLT JointNLT* CTRNLT TNL2K-2 SNLT GTI RTTNLD RVTNLN OSTrack-384 VLTVG
Builds on: Deformable DETR JointNLT Swin Transformer RoBERTa
Closest papers by meaning
- Dynamic Updates for Language Adaptation in Visual-Language Tracking (CVPR 2025 · similarity 0.87)
- ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model (NeurIPS 2024 · similarity 0.82)
- MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models (ICML 2026 · similarity 0.81)
- MambaVLT: Time-Evolving Multimodal State Space Model for Vision-Language Tracking (CVPR 2025 · similarity 0.81)
- Unifying Visual and Vision-Language Tracking via Contrastive Learning (AAAI 2024 · similarity 0.81)
- Introducing Visual Scenes and Reasoning: A More Realistic Benchmark for Spoken Language Understanding (AAAI 2026 · similarity 0.78)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.