Research map / Vision-language and multimodal / Human and embodied modeling
MemVLT: Vision-Language Tracking with Adaptive Memory-based Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MemVLT, a vision-language tracker that uses CLS-inspired short- and long-term memory storage and interaction modules to turn static text/template prompts into adaptive prompts, reaching new SOTA on MGIT, TNL2K, LaSOT and LaSOText.
Problem
Vision-language trackers rely on fixed initial multimodal prompts (template patch and language description), which fail to track targets whose appearance or state changes over time.
Approach
Model human-like memory via complementary learning systems: a section-top long-term memory storage method and a memory interaction module in which short-term visual/textual memories are generated from long-term memory and used to modulate the static prompts into adaptive prompts.
Key results
- Achieves 69.4% AUC on MGIT (action granularity), improving the prior best JointNLT by 8.4%.
- Achieves 63.3% AUC on TNL2K, improving MMTrack by 4.7%.
- Reports 72.9% AUC on LaSOT and 52.1% AUC on LaSOText, outperforming prior vision-language trackers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add memory-based adaptive visual and textual prompts | MemVLT · MGIT action | AUC: 61.0 -> 69.4 | Helped |
| add memory-based adaptive visual and textual prompts | MemVLT · TNL2K | AUC: 58.6 -> 63.3 | Helped |
| add memory-based adaptive visual and textual prompts | MemVLT · LaSOT | AUC: 71.2 -> 72.9 | Helped |
| add memory-based adaptive visual and textual prompts | MemVLT · LaSOText | AUC: 49.4 -> 52.1 | Helped |
| add adaptive visual prompts (ablation) | MemVLT · TNL2K | AUC: 59.0 -> 62.3 | Helped |
| add adaptive textual prompts (ablation) | MemVLT · TNL2K | AUC: 59.0 -> 62.4 | Helped |
| add both adaptive visual and textual prompts (ablation) | MemVLT · TNL2K | AUC: 62.4 -> 63.3 | Helped |
| include global visual and textual tokens in memory construction (ablation) | MemVLT memory interaction module · TNL2K | AUC: 62.1 -> 63.3 | Helped |
| generate adaptive prompts instead of directly integrating search features with short-term memories (ablation) | MemVLT memory interaction module · TNL2K | AUC: 60.9 -> 63.3 | Helped |
| use section-top memory storage rather than sliding window, top-L, or section-L (ablation) | MemVLT memory storage module · TNL2K | AUC: sliding window: 62.3; top-L: 62.5; section-L: 62.8; section-top: 63.3 | Helped |
| vary long-term memory length from 1 to 10 (ablation) | MemVLT · TNL2K | AUC: 1: 62.1; 2: 62.5; 4: 62.8; 6: 63.1; 8: 63.3; 10: 63.1 | Mixed |
| add confidence prediction while keeping sliding-window storage (ablation) | MemVLT · TNL2K | AUC: 62.3 -> 62.3 | No clear effect |
Surprising findings
- Ablation shows that the memory mechanism alone lifts a vision-only variant of the tracker by about 4.3 AUC on TNL2K, suggesting gains come from memory rather than the backbone.
Limitations
- Memory representations lack explicit supervision, reducing interpretability.
- Performance on LaSOT and LaSOText does not surpass the best visual-only trackers, possibly due to vague language descriptions.
Experimental setup
Datasets: LaSOT TNL2K RefCOCOg OTB99-Lang
Benchmarks: MGIT TNL2K LaSOT LaSOText
Metrics: AUC PNorm P
Baselines: JointNLT MMTrack DecoupleTNL VLTTT TransVLT QueryNLT SNLT OVLM All-in-one
Builds on: OSTrack JointNLT AQATrack HiViT RoBERTa
Closest papers by meaning
- ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model (NeurIPS 2024 · similarity 0.90)
- MVLM: Template-Free Tracking via Vision-Language Margin Confidence and Memory-Gated Tracking (CVPR 2026 · similarity 0.87)
- Unifying Visual and Vision-Language Tracking via Contrastive Learning (AAAI 2024 · similarity 0.87)
- Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning (ICML 2024 · similarity 0.87)
- Mono3DVLT: Monocular-Video-Based 3D Visual Language Tracking (CVPR 2025 · similarity 0.87)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.