Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Transductive Zero-Shot and Few-Shot CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes transductive zero-shot and few-shot CLIP classification by modeling CLIP's text-vision probability features on the simplex with Dirichlet distributions, optimized via a block Majorization-Minimization EM-style algorithm over mini-batches of query samples.
Problem
Transductive (batch-joint) inference for CLIP in zero-shot and few-shot image classification has been overlooked, and standard clustering and transductive few-shot methods do not improve over inductive CLIP when applied to vision-language features.
Approach
Represent each sample by CLIP's zero-shot softmax probability vector over classes, model each class's data on the unit simplex with a Dirichlet distribution, and jointly infer assignments and Dirichlet parameters with a closed-form block MM algorithm (EM-Dirichlet) using an MDL partition-complexity penalty.
Key results
- Hard EM-Dirichlet reaches 67.6% average zero-shot accuracy over 11 datasets vs 58.5% for zero-shot CLIP, with ImageNet improving from 58.3% to 77.6% on batches of 75 queries.
- EM-Dirichlet achieves 73.6% average 4-shot accuracy vs 68.3% for Tip-Adapter and 60.9% for PADDLE, outperforming state-of-the-art transductive methods.
- Inductive methods outperform the proposed approach on EuroSAT in the few-shot setting (e.g., Tip-Adapter 72.5% vs 50.8% for EM-Dirichlet).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply transductive Hard EM-Dirichlet classification to zero-shot CLIP probabilities | CLIP ResNet50 · Zero-shot, 11 datasets (average) | accuracy: 58.5 -> 67.6 | Helped |
| Apply transductive Hard EM-Dirichlet classification to zero-shot CLIP probabilities | CLIP ResNet50 · Zero-shot ImageNet | accuracy: 58.3 -> 77.6 | Helped |
| Apply EM-Dirichlet to CLIP probability features | CLIP ResNet50 · Few-shot, 4 shots, 11 datasets (average) | accuracy: 68.3 -> 73.6 | Helped |
| Apply EM-Dirichlet to CLIP probability features | CLIP ResNet50 · Few-shot ImageNet, 4 shots | accuracy: 62.7 -> 78.4 | Helped |
| Use Dirichlet rather than diagonal-covariance Gaussian data fitting (ablation) | EM clustering on CLIP probability features · Zero-shot, 11 datasets (average) | accuracy: 49.7 -> 65.8 | Helped |
| Use text-vision probability features instead of visual embeddings (ablation) | EM-Gaussian (diag cov.) · Zero-shot, 11 datasets (average) | accuracy: 40.4 -> 49.7 | Helped |
| Use text-vision probability features instead of visual embeddings (ablation) | Hard K-means · Zero-shot, 11 datasets (average) | accuracy: 40.6 -> 40.5 | No clear effect |
| Add the entropic assignment term (ablation) | K-means clustering on CLIP probability features · Zero-shot, 11 datasets (average) | accuracy: 40.5 -> 30.3 | Hurt |
| Add MDL partition-complexity regularization to Gaussian EM (ablation) | EM-Gaussian on CLIP probability features · Zero-shot, 11 datasets (average) | accuracy: 19.3 -> 49.7 | Helped |
| Replace Gaussian data fitting with KL-divergence K-means (ablation) | Hard clustering of CLIP probability features · Zero-shot, 11 datasets (average) | accuracy: 40.5 -> 55.6 | Helped |
| Use soft rather than hard assignments (ablation) | EM-Dirichlet · Zero-shot, 11 datasets (average) | accuracy: 67.6 -> 65.8 | Helped |
| Use EM-Dirichlet rather than the strongest transductive baseline PADDLE | CLIP ResNet50 · Few-shot, 4 shots, 11 datasets (average) | accuracy: 60.9 -> 73.6 | Helped |
Surprising findings
- Standard clustering and recent transductive few-shot methods score below their inductive counterparts when applied to CLIP features.
- Few-shot accuracy on Food101 is lower than zero-shot accuracy for the proposed method, consistent with CLIP's observation that few labels can hurt.
Limitations
- Inductive methods outperform the approach on EuroSAT, possibly due to text information confounding.
- Experiments use only the CLIP ResNet50 visual encoder.
- Relies on a fixed temperature T=30 and a fixed lambda setting rather than per-dataset tuning in zero-shot.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101
Metrics: accuracy execution time
Baselines: Tip-Adapter CoOp BD-CSPN Laplacian Shot α-TIM PADDLE Hard K-means Soft K-means EM-Gaussian Hard KL K-means
Builds on: CLIP PADDLE Minka Dirichlet MLE TIP-Adapter soft K-means
Closest papers by meaning
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.89)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.88)
- Conformal Prediction for Zero-Shot Models (CVPR 2025 · similarity 0.88)
- Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting (NeurIPS 2024 · similarity 0.87)
- UNEM: UNrolled Generalized EM for Transductive Few-Shot Learning (CVPR 2025 · similarity 0.87)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.