Research map / Vision-language and multimodal / CLIP and zero-shot
ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reframes Tip-Adapter's cache model as a Nadaraya-Watson kernel estimator and proposes ProKeR, a training-free closed-form proximal kernel ridge regression with CLIP as base learner that sets new training-free few-shot SOTA on 11 datasets.
Problem
Caching-based training-free few-shot adaptation of CLIP (e.g., Tip-Adapter) lacks theoretical understanding and global regularization from the zero-shot predictor, leading to bias and limited generalization.
Approach
Views Tip-Adapter's cache term as a Nadaraya-Watson estimator, then fixes its bias with local linear regression and a Mahalanobis metric, and finally introduces a proximal regularizer in an RKHS (kernel ridge regression toward the CLIP zero-shot predictor) that gives a closed-form global solution.
Key results
- ProKeR (RBF) reaches 69.07% average accuracy over 1-16 shots on 11 datasets with ImageNet-transferred hyperparameters, versus 61.68% for Tip-Adapter and 67.03% for GDA.
- Under the CoOp benchmark protocol, ProKeR achieves 70.46% average vs. 66.56% for Tip-Adapter and 69.49% for GDA.
- ProKeR+CLAP reaches 70.73% average, outperforming training-based CLAP (68.92%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add global proximal regularization in an RKHS to Tip-Adapter | CLIP few-shot adaptation · 11 classification datasets, validation-free, 16 shots | average accuracy: 64.61 -> 75.12 | Helped |
| Use a Polynomial kernel instead of an RBF kernel (ablation) | ProKeR · 11 classification datasets, CoOp benchmark, 16 shots | accuracy: 76.75 -> 76.61 | Hurt |
| Use an Epanechnikov kernel instead of an RBF kernel (ablation) | ProKeR · 11 classification datasets, CoOp benchmark, 16 shots | accuracy: 76.75 -> 74.43 | Hurt |
| Use a Linear kernel instead of an RBF kernel (ablation) | ProKeR · 11 classification datasets, CoOp benchmark, 16 shots | accuracy: 76.75 -> 72.34 | Hurt |
| Approximate the RBF kernel with Random Fourier features (ablation) | ProKeR · 11 classification datasets, CoOp benchmark, 16 shots | average accuracy: 76.75 -> 76.44 | Hurt |
| Jointly train ProKeR with CLAP (ablation) | ProKeR · 11 classification datasets, validation-free, 16 shots | average accuracy: 75.12 -> 76.84 | Helped |
| Replace the local constant fit with regularized local linear regression (ablation) | Tip-Adapter · 11 image classification datasets | average accuracy | Helped |
| Use a Mahalanobis distance as the global metric (ablation) | NW estimator · 11 image classification datasets | average accuracy | Helped |
| Use the query-dependent, training-size-agnostic NW reformulation (ablation) | Tip-Adapter · 11 image classification datasets, 8 and 16 shots | average accuracy | Helped |
Surprising findings
- Tip-Adapter's cache term is equivalent to a Nadaraya-Watson estimator, and its closed-form regularized variant is agnostic to training set size.
- Polynomial and linear kernels underperform RBF on the kernel ablation, though the polynomial kernel is competitive with RBF at 16 shots.
Limitations
- Global regularization and local flexibility are not yet combined; authors note the trade-off between global and local methods on complex data.
- Hyperparameters in the CoOp-protocol setting are tuned on the test set, which the authors note may be impractical.
- Training-based combination (ProKeR+CLAP) is not training-free and increases cost.
Experimental setup
Datasets: ImageNet Caltech101 DTD EuroSAT FGVCAircraft Flowers102 Food101 OxfordPets StanfordCars SUN397 UCF101
Benchmarks: ImageNet Caltech101 DTD EuroSAT FGVCAircraft Flowers102 Food101 OxfordPets StanfordCars SUN397 UCF101 ImageNet-V2
Metrics: accuracy average accuracy running time
Baselines: Tip-Adapter APE GDA Tip-Adapter-F CrossModalLP TaskRes CLAP CALIP Tip-X CoOp CLIP-Adapter
Builds on: Tip-Adapter CLAP APE Nadaraya-Watson estimator CLIP
Closest papers by meaning
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.90)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.90)
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.89)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Adaptive Parameter Selection for Tuning Vision-Language Models (CVPR 2025 · similarity 0.89)
- Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.