Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits Gaussian Discriminant Analysis with shared covariance to build a training-free CLIP classifier from class means and a shrinkage precision matrix, ensembled with the zero-shot text classifier, extended to base-to-new and unsupervised settings.
Problem
Efficient adaptation of CLIP to downstream classification without the extra training time and compute required by prompt learning and adapter methods, which is a problem for resource-limited devices.
Approach
Treat CLIP visual features as class-conditional Gaussians with a shared covariance, so the Bayes classifier reduces to a linear function of class means and a precision matrix estimated in closed form (empirical Bayes ridge-type shrinkage), then ensemble it with CLIP's zero-shot text classifier. No gradient training is needed, and only the classifier parameters are stored, so it scales to full trai
Key results
- On 11 datasets at 16 shots with RN50 CLIP, the method averages 76.05% vs 75.83% for training-required Tip-Adapter-F and 73.23% for the best training-free baseline APE.
- On ImageNet-LT the method reaches 62.34% overall accuracy versus 60.97% for full fine-tune MARC, and 42.07% overall on Places-LT.
- Ensembling the GDA classifier with zero-shot CLIP outperforms either component alone across all shot settings, and the method trains on full ImageNet in 3.6 seconds on a single RTX 3090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the GDA classifier ensemble instead of the strongest training-required baseline | CLIP few-shot classification · 11-dataset average, 16-shot | accuracy: 75.83 -> 76.05 | Helped |
| Use the GDA classifier ensemble instead of APE | CLIP few-shot classification · 11-dataset average, 16-shot | accuracy: 2.82% | Helped |
| Apply GDA adaptation | CLIP · ImageNet-LT and Places-LT | overall accuracy: 8.72% and 9.90% | Helped |
| Estimate GDA means and covariance with EM on unlabeled data | CLIP unsupervised learning · 11-dataset average | accuracy: 4.69% | Helped |
| Synthesize new-class examples with KNN and apply GDA | CLIP base-to-new generalization · Average over 11 datasets | harmonic mean: 7.02%, 7.06%, 2.89%, and 1.72% | Helped |
| Replace KS precision estimation with LedoitWolf (ablation) | GDA adaptation · EuroSAT, 16-shot | accuracy: 86.12 -> 84.33 | Hurt |
| Set ensemble coefficient alpha to 1 (ablation) | GDA plus zero-shot CLIP · 11-dataset few-shot average | accuracy: 75.49 -> 73.45 | Hurt |
| Set ensemble coefficient alpha to 3 (ablation) | GDA plus zero-shot CLIP · 11-dataset few-shot average | accuracy: 73.45 -> 75.49 | Helped |
Surprising findings
- The standalone linear GDA classifier sometimes performs worse than the zero-shot classifier, and only the ensemble consistently beats both.
- Tip-Adapter and Tip-Adapter-F run out of memory when using the full ImageNet training set because they cache all training features, while the GDA method stores only the classifier parameters.
Limitations
- Base-to-new and unsupervised variants are simple extensions and only comparable to, not clearly better than, prior state-of-the-art methods.
- The mixing hyperparameter alpha is tuned on a validation set with a coarse grid.
- The base-to-new variant relies on a KNN proxy for new-class data because the method cannot be applied directly without it.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft EuroSAT UCF101 DTD SUN397 ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft EuroSAT UCF101 DTD SUN397 ImageNetV2
Metrics: accuracy macro F1 harmonic mean
Baselines: Zero-Shot CLIP CALIP Tip-Adapter Tip-X APE linear probe CoOp CLIP-Adapter Tip-Adapter-F CoCoOp KgCoOp UPL
Builds on: CLIP CoOp Tip-Adapter Gaussian Discriminant Analysis
Closest papers by meaning
- AmorLIP: Efficient Language-Image Pretraining via Amortization (NeurIPS 2025 · similarity 0.90)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.90)
- CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation (NeurIPS 2025 · similarity 0.90)
- OT-CLIP: Understanding and Generalizing CLIP via Optimal Transport (ICML 2024 · similarity 0.90)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.90)
- CLIP-KD: An Empirical Study of CLIP Model Distillation (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.