Research map / Vision-language and multimodal / CLIP and zero-shot
Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dual Prototype Evolving (DPE), a training-light test-time adaptation method for CLIP that evolves textual and visual prototypes across a test stream and optimizes per-sample prototype residuals with entropy and alignment losses.
Problem
Test-time adaptation of vision-language models like CLIP typically adapts from only one modality and treats each test sample independently, without accumulating task-specific knowledge over the test stream.
Approach
Maintain two evolving prototype sets: textual prototypes updated by cumulative averaging over confident samples, and visual prototypes stored in per-class priority queues of low-entropy image features. Per-sample learnable residuals on both prototype sets are optimized with entropy minimization over augmented views plus an InfoNCE-style cross-modal alignment loss, without backpropagating through t
Key results
- On natural distribution shifts with ResNet-50, DPE reaches 50.81% average vs. 47.26% for TPT (+3.55%) and 49.58% for TDA.
- On cross-dataset generalization with ViT-B/16, DPE averages 69.40% across 10 datasets vs. 67.53% for TDA and 65.10% for TPT.
- Efficiency on ImageNet: DPE takes 1 h 50 min vs. 9 h 15 min for TPT, roughly 5x faster, though slower than TDA (1 h 5 min).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DPE instead of the strongest listed baseline, DMN-ZS | CLIP ResNet-50 · ImageNet natural shifts (Average) | top-1 accuracy (%): 49.97 -> 50.81 | Helped |
| use DPE instead of the strongest listed baseline, DMN-ZS | CLIP ViT-B/16 · ImageNet natural shifts (Average) | top-1 accuracy (%): 65.49 -> 65.93 | Helped |
| use DPE instead of the strongest listed baseline, TDA | CLIP ResNet-50 · cross-dataset generalization average | top-1 accuracy (%): 61.03 -> 61.93 | Helped |
| use DPE instead of the strongest listed baseline, TDA | CLIP ViT-B/16 · cross-dataset generalization average | top-1 accuracy (%): 67.53 -> 69.40 | Helped |
| use cumulative average rather than no textual prototype update (ablation) | DPE · ImageNet | accuracy: 62.93 -> 63.41 | Helped |
| replace textual prototypes with the per-sample optimized prototypes (ablation) | DPE · ImageNet | accuracy: 62.93 -> 21.83 | Hurt |
| use exponential average with decay rate 0.99 instead of no textual prototype update (ablation) | DPE · ImageNet | accuracy: +0.18% | Helped |
| use exponential average with decay rate 0.95 instead of no textual prototype update (ablation) | DPE · ImageNet | accuracy: -0.36% | Hurt |
| add visual prototype residual learning (ablation) | DPE · ImageNet | accuracy: +0.36% | Helped |
| add textual prototype residual learning (ablation) | DPE · ImageNet | accuracy: +1.40% | Helped |
| add both textual and visual prototype residual learning (ablation) | DPE · ImageNet | accuracy: +1.52% | Helped |
| add the alignment loss to entropy minimization (ablation) | DPE · three datasets | accuracy: 0.23% to 1.07% | Helped |
Surprising findings
- Fully replacing textual prototypes with per-sample optimized prototypes collapses ImageNet accuracy to 21.83%, so the evolution must be cumulative and gradual.
- Reported DiffTPT results are on a 1,000-sample subset of the datasets, which the authors note may introduce imprecision.
Limitations
- Still requires gradient backpropagation through the prototype residuals, adding per-sample compute over zero-shot CLIP.
- Priority queues increase memory cost at inference time.
- Baseline results were copied from original papers rather than re-run, which the authors note is for fairness.
Experimental setup
Datasets: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch Caltech101 DTD EuroSAT FGVCAircraft Flowers102 Food101 OxfordPets
Benchmarks: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch FGVCAircraft Caltech101 StanfordCars DTD EuroSAT Flowers102 Food101
Metrics: top-1 accuracy inference time
Baselines: Zero-shot CLIP Ensemble CoOp TPT DiffTPT TDA TPS DMN-ZS
Builds on: CLIP test-time prompt tuning Tip-Adapter TDA
Closest papers by meaning
- The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- Realistic Test-Time Adaptation of Vision-Language Models (CVPR 2025 · similarity 0.91)
- Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts (NeurIPS 2026 · similarity 0.90)
- Efficient Test-Time Adaptation of Vision-Language Models (CVPR 2024 · similarity 0.90)
- DOTA: Distributional Test-time Adaptation of Vision-Language Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.