Research map / Label-efficient and robust learning / Test-time adaptation
Any-Shift Prompting for Generalization over Distributions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes any-shift prompting, a hierarchical probabilistic prompt-learning framework on frozen CLIP that generates a test-specific prompt in one feedforward pass via a transformer inference network trained with pseudo-shift mini-batches.
Problem
Prompt learning on CLIP overfits the training distribution and fails to generalize under covariate, label, concept, conditional, and joint distribution shifts at test time.
Approach
Model training and test prompts as latent variables in a hierarchical Bayesian framework, with a shared transformer inference network that infers a test prompt (prior from one test image and class names; posterior from a batch with labels) and is trained with pseudo-shift mini-batches so no test-time optimization is needed.
Key results
- On covariate shift (eight datasets), the method reaches 98.16 on PACS and 60.93 on DomainNet, outperforming CoCoOp on all eight datasets.
- On Open-Office-Home (joint shift), mean accuracy 84.50 vs 82.19 for CoCoOp and 80.35 for CLIP.
- Concept shift on ImageNet-Superclass improves to 71.12 from CLIP's 69.23; Entity-30 conditional shift reaches 81.74 vs 67.95 for CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Any-Shift Prompting instead of DPL | CLIP ViT-B/16 · PACS | accuracy: 98.16 ± 0.4 vs 97.07 | Helped |
| use Any-Shift Prompting instead of DPL | CLIP ViT-B/16 · VLCS | accuracy: 86.54 ± 0.4 vs 83.99 | Helped |
| use Any-Shift Prompting instead of DPL | CLIP ViT-B/16 · Office-Home | accuracy: 85.16 ± 0.6 vs 83.00 | Helped |
| use Any-Shift Prompting instead of DPL | CLIP ViT-B/16 · DomainNet | accuracy: 60.93 ± 0.6 vs 59.86 | Helped |
| use Any-Shift Prompting instead of CoCoOp | CLIP ViT-B/16 · ImageNet-Superclass | accuracy: 71.12 ± 0.6 vs 69.77 | Helped |
| use Any-Shift Prompting instead of CoCoOp | CLIP ViT-B/16 · Living-17 | accuracy: 88.41 ± 0.3 vs 87.24 | Helped |
| use Any-Shift Prompting instead of CoCoOp | CLIP ViT-B/16 · Entity-30 | accuracy: 81.74 ± 0.4 vs 79.52 | Helped |
| use Any-Shift Prompting instead of CoCoOp | CLIP ViT-B/16 · Open-Office-Home | mean accuracy: 84.50 ± 0.4 vs 82.19 | Helped |
| use only the training prompt (ablation) | Any-Shift Prompting · Open-Office-Home | accuracy: 82.62 vs 84.50 | Hurt |
| use only test text features (ablation) | Any-Shift Prompting · Open-Office-Home | accuracy: 82.67 vs 84.50 | Hurt |
| use only test image features (ablation) | Any-Shift Prompting · Open-Office-Home | accuracy: 83.11 vs 84.50 | Hurt |
| remove the training prompt from the test text-and-image feature setting (ablation) | Any-Shift Prompting · Open-Office-Home | accuracy: 83.63 vs 84.50 | Hurt |
Limitations
- Relies on CLIP frozen encoders, so performance bounded by CLIP's pretrained knowledge
- Requires labeled training data from the downstream task to learn the inference network
Experimental setup
Datasets: PACS VLCS Office-Home DomainNet ImageNet ImageNet-V2 ImageNet-Sketch ImageNet-A ImageNet-R Caltech101 OxfordPets StanfordCars
Benchmarks: PACS VLCS Office-Home DomainNet ImageNet-V2 ImageNet-Sketch ImageNet-A ImageNet-R 11 base-to-new datasets ImageNet-Superclass Living-17 Entity-30
Metrics: classification accuracy harmonic mean
Baselines: CLIP CLIP-D CoOp CoCoOp DPL BPL TPT MaPLe
Builds on: CLIP CoOp CoCoOp TPT Bayesian prompt learning
Closest papers by meaning
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Understanding prompt engineering may not require rethinking generalization (ICLR 2024 · similarity 0.89)
- Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP (NeurIPS 2024 · similarity 0.89)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.89)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.88)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.