Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Bayesian Exploration of Pre-trained Models for Low-shot Image Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Gaussian process ensemble for low-shot image classification that uses zero-shot CLIP as the GP mean and a sum of deep kernels from CLIP, MoCo and DINO as the kernel, with hyper-parameters tuned by predictive likelihood.
Problem
Low-shot image classification with CLIP-based methods lacks flexibility to integrate other pre-trained models, and existing ensembles (e.g., CaFo) use heuristic weights and produce poorly calibrated, overfit deterministic predictions.
Approach
Use Gaussian process label regression where the prior mean is the zero-shot CLIP classifier and the kernel is a sum of learnable-lengthscale RBF deep kernels built on frozen pre-trained encoders, giving analytic posterior, uncertainty, and principled hyper-parameter tuning.
Key results
- On ImageNet 16-shot, the method reaches 70.77% accuracy versus 69.83% for the strongest ensemble baseline (Ens-LP†) and 65.81% for CALIP-FS.
- AUROC for distinguishing ImageNet from ImageNet-Sketch is 0.8545, above Ens-LP† (0.8253) and Ens-CaFo (0.7546).
- TACE is 0.0169 on ImageNet, the best among compared ensembles, though its ECE (0.0786) is slightly worse than Ens-CaFo (0.0577).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Ensemble complementary pre-trained-model kernels and use a zero-shot CLIP prior mean instead of Ens-CaFo | Low-shot image classification · ImageNet, 1-shot | accuracy (%): 62.09 ± 0.13 -> 63.07 | Helped |
| Ensemble complementary pre-trained-model kernels and use a zero-shot CLIP prior mean instead of Ens-LP† | Low-shot image classification · ImageNet, 2-shot | accuracy (%): 64.10 ± 0.35 -> 65.17 | Helped |
| Ensemble complementary pre-trained-model kernels and use a zero-shot CLIP prior mean instead of Ens-LP† | Low-shot image classification · ImageNet, 4-shot | accuracy (%): 65.89 ± 0.33 -> 67.50 | Helped |
| Ensemble complementary pre-trained-model kernels and use a zero-shot CLIP prior mean instead of Ens-LP† | Low-shot image classification · ImageNet, 8-shot | accuracy (%): 67.59 ± 0.06 -> 69.31 | Helped |
| Ensemble complementary pre-trained-model kernels and use a zero-shot CLIP prior mean instead of Ens-LP† | Low-shot image classification · ImageNet, 16-shot | accuracy (%): 69.83 ± 0.27 -> 70.77 | Helped |
| Use the proposed pre-trained-model ensemble instead of CALIP-FS | Low-shot image classification · ImageNet, 1–16 shots | accuracy (%) | Helped |
| Use the proposed method instead of Ens-CaFo | Low-shot image classification · ImageNet-V2 and ImageNet-Sketch | accuracy (%) | Helped |
| Use the proposed method instead of Ens-CaFo | Image classification · ImageNet | TACE: 0.0545 -> 0.0169 | Helped |
| Use the proposed method instead of Ens-CaFo | Image classification · ImageNet | ECE: 0.0577 -> 0.0786 | Hurt |
| Replace the zero-shot CLIP predictor prior mean with a constant-zero or learnable-vector mean (ablation) | Proposed GP classifier · ImageNet low-shot classification | accuracy | Hurt |
| Replace the RBF base kernel with Laplacian or Matérn (ablation) | Proposed GP classifier · ImageNet low-shot classification | accuracy | Hurt |
| Add multiple pre-trained-model kernels to the CLIP kernel (ablation) | Proposed GP classifier · ImageNet low-shot classification | accuracy | Helped |
Surprising findings
- Ens-LP improves greatly when the zero-shot CLIP classifier is added (Ens-LP 66.31% vs Ens-LP† 69.83% at 16 shots), highlighting the value of the zero-shot prior.
- Marginal likelihood tuning underperforms predictive likelihood tuning, contrary to the common default.
Limitations
- ECE is slightly worse than Ens-CaFo, so calibration advantage depends on the metric (TACE).
- Only ResNet-50 pre-trained encoders are used in experiments; scaling to other backbones is claimed but not tested.
- Compute is reported only for hyper-parameter optimization on a single RTX-3090.
Experimental setup
Datasets: ImageNet StanfordCars UCF101 Caltech101 Flowers102 SUN397 DTD EuroSAT FGVCAircraft OxfordPets Food101
Benchmarks: ImageNet StanfordCars UCF101 Caltech101 Flowers102 SUN397 DTD EuroSAT FGVCAircraft OxfordPets Food101 ImageNet-V2
Metrics: top-1 accuracy ECE TACE AUROC
Baselines: Ens-LP Ens-LP† Ens-CaFo CoOp CLIP-Adapter VT-CLIP Tip-Adapter-F CALIP-FS
Builds on: CaFo Tip-Adapter deep kernel learning CLIP
Closest papers by meaning
- Bayesian Low-Rank Learning (Bella): A Practical Approach to Bayesian Neural Networks (AAAI 2025 · similarity 0.87)
- Transductive Zero-Shot and Few-Shot CLIP (CVPR 2024 · similarity 0.87)
- Accelerating Convergence in Bayesian Few-Shot Classification (ICML 2024 · similarity 0.87)
- Bayesian Test-Time Adaptation for Vision-Language Models (CVPR 2025 · similarity 0.86)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.86)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.