Research map / Vision-language and multimodal / Human and embodied modeling
ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ChatTracker, which uses an MLLM (GPT-4V) with a reflection-based prompt optimization loop, grounding-model feedback, and a semantic tracking/verification module to generate target and background text descriptions that boost visual and vision-language trackers.
Problem
Vision-language trackers underperform SoTA visual trackers, partly because manual textual annotations are ambiguous and inaccurate; directly using MLLM-generated descriptions does not improve tracking because of a knowledge gap with the tracker and MLLM hallucination.
Approach
Iteratively refine MLLM-generated foreground/background descriptions using feedback from a grounded VL model (GroundingDINO) on the template frame: words are labeled positive or negative by whether they ground to the target, and this feedback is fed back to the MLLM as a reflection prompt. Descriptions are then used for semantic tracking with a foreground verification step that combines template s
Key results
- ChatTracker-L reaches 74.1% AUC on LaSOT, surpassing JointNLT by 13.5 points, and 86.1% AUC on TrackingNet.
- ChatTracker-L achieves 65.4% AUC and 70.2% precision on TNL2K, with precision 12.1 points above JointNLT.
- Plugging ChatTracker into OSTrack-256, TransT-N4, and Stark-S improves LaSOT AUC by about 1.1 to 2.5 points per tracker (e.g., TransT-N4 64.85 to 67.34).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| integrate ChatTracker-L with MixFormer | MixFormer · LaSOT | AUC: 74.1 vs 72.6 | Helped |
| integrate ChatTracker-L with MixFormer | MixFormer · TrackingNet | AUC: 86.1 vs 85.5 | Helped |
| integrate ChatTracker-L with MixFormer | MixFormer · TNL2K | AUC: 65.4 vs 63.1 | Helped |
| add ChatTracker to OStrack-256 | OStrack-256 · LaSOT | AUC: 69.11 -> 70.23 | Helped |
| add ChatTracker to TransT-N4 | TransT-N4 · TNL2K | AUC: 53.16 -> 56.27 | Helped |
| replace manual text with ChatTracker-generated text | JointNLT · LaSOT | AUC: 56.74 -> 57.96 | Helped |
| replace manual text with ChatTracker-generated text | UVLTrack · TNL2K | AUC: 54.78 -> 55.61 | Helped |
| use Gemini1.0 instead of GPT-4V | ChatTracker-B · TNL2K | AUC: 59.63 -> 60.23 | Helped |
| use LLaVA-7B instead of GPT-4V | ChatTracker-B · LaSOT | AUC: 71.68 -> 71.36 | Hurt |
| add RPO, semantic tracking, and foreground verification to the visual-only base (ablation) | TransT-N4 Base Model · LaSOT | AUC: 64.85 -> 67.89 | Helped |
| add RPO, semantic tracking, and foreground verification to the visual-only base (ablation) | TransT-N4 Base Model · TNL2K | AUC: 53.16 -> 56.39 | Helped |
| use manual text and omit RPO-generated background descriptions and background scoring (ablation) | ChatTracker · LaSOT | AUC: 67.89 -> 64.63 | Hurt |
Surprising findings
- Using manually annotated text from datasets yields performance similar to the base visual tracker (w/o RPO), suggesting manual descriptions are sub-optimal for tracking.
- The open-source LLaVA-7B performs comparably to proprietary GPT-4V and Gemini when plugged into ChatTracker-B.
Limitations
- MLLM struggles with low-resolution targets or targets lacking discernible visual features.
- Depends on API access to MLLM, requiring internet connectivity for edge deployment.
- Language descriptions are not updated during tracking, so temporal appearance changes are not handled.
Experimental setup
Datasets: LaSOT TrackingNet TNL2K OTB-lang
Benchmarks: LaSOT TrackingNet TNL2K OTB-lang
Metrics: AUC precision normalized precision text-to-image alignment score
Baselines: JointNLT CiteTracker DecoupleTNL UVLTrack-B RGFM MixformerV2 F-BDMTrack MITS ARTrack SeqTrack DropTrack MATTracker
Builds on: GroundingDINO CLIP ARTrack MixFormer UVLTrack
Closest papers by meaning
- MemVLT: Vision-Language Tracking with Adaptive Memory-based Prompts (NeurIPS 2024 · similarity 0.90)
- Unifying Visual and Vision-Language Tracking via Contrastive Learning (AAAI 2024 · similarity 0.89)
- MVLM: Template-Free Tracking via Vision-Language Margin Confidence and Memory-Gated Tracking (CVPR 2026 · similarity 0.87)
- Mono3DVLT: Monocular-Video-Based 3D Visual Language Tracking (CVPR 2025 · similarity 0.87)
- VastTrack: Vast Category Visual Object Tracking (NeurIPS 2024 · similarity 0.86)
- MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.