Research map / Label-efficient and robust learning / Test-time adaptation
Historical Test-time Prompt Tuning for Vision Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HisTPT, a test-time prompt tuning framework that uses three memory banks (local, hard-sample, global) plus adaptive retrieval to mitigate forgetting when prompts are updated on a continuous unlabelled test stream.
Problem
Test-time prompt tuning of vision foundation models degrades over long online tuning, especially when test domains change continuously, because self-supervised prompt updates accumulate errors and forget earlier useful knowledge.
Approach
Stores features and predictions of previously seen test samples in three complementary knowledge banks (recent-sample FIFO queue, high-entropy hard-sample queue, and a momentum-updated global prototype) and adaptively retrieves them, weighted by prediction entropy, to regularize each new test sample's prediction and prompt update.
Key results
- On semantic segmentation with SEEM-Tiny, HisTPT reaches 33.5 mean mIoU vs 31.9 for TPT and 32.3 for DiffTPT over 6 datasets (ACDC subsets included).
- On 10 classification datasets with CLIP ViT-B/16, HisTPT reaches 67.6 mean top-1 vs 64.6 for TPT and 65.0 for DiffTPT.
- Ablation on Cityscapes with SEEM-Tiny: all three banks plus adaptive retrieval reach 44.7 mIoU vs 39.2 for the base model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply HisTPT rather than DiffTPT | SEEM-Tiny · Semantic segmentation (6-dataset mean) | mIoU: 32.3 -> 33.5 | Helped |
| Apply HisTPT rather than DiffTPT | CLIP-ViT-B/16 · Image classification (10-dataset mean) | top-1 accuracy: 65.0 -> 67.6 | Helped |
| Apply HisTPT rather than DiffTPT | SEEM-Large · Object detection (4-dataset mean) | mAP50: 36.0 -> 36.8 | Helped |
| Add the local knowledge bank (ablation) | HisTPT on SEEM-Tiny · Cityscapes semantic segmentation | mIoU: 39.2 -> 41.1 | Helped |
| Add the hard-sample knowledge bank (ablation) | HisTPT on SEEM-Tiny · Cityscapes semantic segmentation | mIoU: 39.2 -> 40.9 | Helped |
| Add the global knowledge bank (ablation) | HisTPT on SEEM-Tiny · Cityscapes semantic segmentation | mIoU: 39.2 -> 41.7 | Helped |
| Add adaptive knowledge retrieval to all three knowledge banks (ablation) | HisTPT on SEEM-Tiny · Cityscapes semantic segmentation | mIoU: 43.6 -> 44.7 | Helped |
| Initialize HisTPT with CoOp prompts | HisTPT · Image classification (10-dataset mean) | top-1 accuracy: 61.2 -> 62.4 | Helped |
| Increase the optimization steps from 1 toward 10 | HisTPT with SEEM-Tiny · Semantic segmentation (6-dataset mean) | mIoU | Helped |
| Tune prompts continuously across changing weather domains | TPT · Cityscapes semantic segmentation, normal-to-adverse order | mIoU: 42.3(+3.1) -> 34.8(+0.2) -> 20.1(-0.6) -> 31.7(-1.4) -> 30.6(-5.2) | Hurt |
Surprising findings
- Existing test-time prompt tuning methods (TPT, DiffTPT) learn good prompts early but their performance falls below the initial template prompt as tuning continues over the test stream.
- Learnt prompts from CoOp/CoCoOp used as initialization for HisTPT further improve results, suggesting complementarity with training-time prompt learning.
Limitations
- Gains on some datasets are modest, and results depend on the chosen bank sizes and hyperparameters (L, H, K, gamma).
- Segmentation and detection baselines are reproduced by the authors from classification methods, since little prior work exists for those tasks.
Experimental setup
Datasets: Cityscapes BDD100K Mapillary ADE20K Pascal Context ACDC Flowers102 DTD Oxford-Pets StanfordCars UCF101 Caltech101
Benchmarks: Cityscapes BDD100K Mapillary ADE20K Pascal Context ACDC Flowers102 DTD Oxford-Pets StanfordCars UCF101 Caltech101
Metrics: mIoU top-1 accuracy mAP50
Baselines: TPT DiffTPT HCL MeGA BiMem MeaCap TF-Clip TDA CoOp CoCoOp
Builds on: TPT DiffTPT CoOp CoCoOp SEEM
Closest papers by meaning
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.89)
- Hierarchical Knowledge Prompt Tuning for Multi-task Test-Time Adaptation (CVPR 2025 · similarity 0.89)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.88)
- DynaPrompt: Dynamic Test-Time Prompt Tuning (ICLR 2025 · similarity 0.88)
- Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning? (ICLR 2024 · similarity 0.88)
- Test-Time Visual In-Context Tuning (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.