Research map / Label-efficient and robust learning / Test-time adaptation
Hierarchical Knowledge Prompt Tuning for Multi-task Test-Time Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HKPT, a teacher-student CLIP prompt distillation framework with task-specific dual dynamic knowledge graphs and adaptive task grouping for joint multi-task test-time adaptation, also usable in single-task TTA.
Problem
Test-time prompt tuning of vision-language models like CLIP is designed for a single target domain and loses effectiveness when adapting jointly to multiple target tasks from unlabeled test streams.
Approach
Distills broader source knowledge from a larger teacher CLIP (ViT-L/14) into a lightweight student CLIP (ViT-B/16) via logits and feature distillation, models task-specific context with per-task dual (visual and textual) dynamic knowledge graphs linked by virtual nodes, and shares knowledge across tasks through adaptively learned task groups.
Key results
- On MiniDomainNet multi-task setting, HKPT reaches 87.7% average accuracy vs 86.1% for MmAP; on Office-Home 87.3% vs 86.5%.
- In single-task setting, HKPT averages 69.7% on the cross-domain benchmark (vs 67.5% for TDA) and 67.3% on the OOD benchmark (vs 65.0% for TDA).
- Ablation on MiniDomainNet shows removing the task-shared prompt drops average accuracy by 6.8 points and removing cross-task knowledge sharing drops it by 2.0 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HKPT instead of MmAP | Multi-task test-time adaptation · Office-Home average | average accuracy (%): +0.8% | Helped |
| Use HKPT instead of MmAP | Multi-task test-time adaptation · MiniDomainNet average | average accuracy (%): +1.6% | Helped |
| Use HKPT instead of TDA | Single-task test-time adaptation · Cross-domain benchmark average | accuracy (%): +2.2% | Helped |
| Use HKPT instead of TDA | Single-task test-time adaptation · OOD benchmark average | accuracy (%): +2.3% | Helped |
| Add a visual knowledge subgraph (ablation) | HKPT · MiniDomainNet | average accuracy (%): 85.1 -> 87.7 | Helped |
| Add a textual knowledge subgraph (ablation) | HKPT · MiniDomainNet | average accuracy (%): 86.4 -> 87.7 | Helped |
| Add virtual nodes connecting the visual and textual subgraphs (ablation) | HKPT · MiniDomainNet | average accuracy (%): 86.7 -> 87.7 | Helped |
| Add the dual knowledge graph module (ablation) | HKPT · MiniDomainNet | average accuracy (%): 83.6 -> 87.7 | Helped |
| Add a learnable task-shared prompt (ablation) | HKPT · MiniDomainNet | average accuracy (%): 80.9 -> 87.7 | Helped |
| Add feature-level distillation (ablation) | HKPT · MiniDomainNet | average accuracy (%): 86.5 -> 87.7 | Helped |
| Add the cross-task knowledge sharing module (ablation) | HKPT · MiniDomainNet | average accuracy (%): 85.7 -> 87.7 | Helped |
Surprising findings
- HKPT still adapts acceptably even when the teacher CLIP is weaker than the student (ViT-B/32 teacher case).
Limitations
- Multi-task grouping relies on exhaustive enumeration of grouping strategies, which may scale poorly with many tasks.
- Hyperparameters (entropy threshold, number of groups, alpha, beta) are fixed per setting and their sensitivity is not broadly explored.
Experimental setup
Datasets: Office-Home MiniDomainNet ImageNet ImageNet-V2 ImageNet-A ImageNet-R ImageNet-S Aircraft Caltech101 DTD Flowers102 Oxford-Pets
Benchmarks: Office-Home MiniDomainNet ImageNet OOD benchmark (ImageNet, -A, -V2, -R, -S) cross-domain benchmark (10 datasets)
Metrics: top-1 accuracy average accuracy
Baselines: CLIP zero-shot Full FT CoOp VPT MaPLe CLIP-Adapter BitFit MmAP CoCoOp TPT DiffTPT C-TPT
Builds on: CLIP TPT TDA MmAP CoOp
Closest papers by meaning
- Historical Test-time Prompt Tuning for Vision Foundation Models (NeurIPS 2024 · similarity 0.89)
- DART: Dual-Modal Adaptive Online Prompting and Knowledge Retention for Test-Time Adaptation (AAAI 2024 · similarity 0.88)
- Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning (ICLR 2025 · similarity 0.87)
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.87)
- Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models (AAAI 2024 · similarity 0.87)
- DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.