Research map / Vision-language and multimodal / CLIP and zero-shot
Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Frolic, a training-free, label-free framework that models CLIP class prototypes as Gaussians fitted from unlabeled test-data moments, fuses them with zero-shot CLIP via confidence matching, and corrects pre-training label bias with an unsupervised logit adjustment.
Problem
Zero-shot CLIP performance is limited by fixed text prototypes that ignore visual variation and by label bias from imbalanced web-scale pre-training data; existing fixes need labeled downstream data or inaccessible pre-training data.
Approach
Infer class-conditional Gaussian covariance from the first and second moments of unlabeled data (no labels), fuse the Gaussian classifier with CLIP by matching average confidences instead of tuning a mixing weight, and estimate the pre-training label prior from downstream unlabeled data via a linear eigen-equation (S-I)β=0 solved by power iteration.
Key results
- On 10 datasets with CLIP ViT-B/16, Frolic reaches 71.1% average accuracy versus 68.6% for the best prior (CuPL-CLIP), and 72.7% when combined with InMaP.
- On ImageNet and five shifts with ViT-B/16, Frolic averages 64.4%, above CuPL-CLIP at 62.7% and CLIP at 60.1%.
- Ablation on 10 datasets with ViT-B/16: the adaptive fusion row reaches 70.4% vs 66.3% for simple fusion, and the bias-corrected fused model reaches 71.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Model class prototypes with a Gaussian prompt distribution instead of single CLIP prototypes (ablation) | CLIP · 10-datasets, ViT-B/16 | accuracy: 65.1 -> 68.8; 3.7% accuracy improvement | Helped |
| Replace direct logit addition with confidence-matched adaptive fusion (ablation) | CLIP + Gaussian prompt distribution model · 10-datasets, ViT-B/16 | accuracy: 66.3 -> 70.4 | Helped |
| Add logit adjustment for label-bias correction (ablation) | Adaptive fusion f_f · 10-datasets, ViT-B/16 | accuracy: 70.4 -> 71.1 | Helped |
| Add logit adjustment for label-bias correction (ablation) | CLIP f_c · 10-datasets, ViT-B/16 | accuracy: 65.1 -> 68.4 | Helped |
| Apply Frolic to InMaP | InMaP, CLIP ViT-B/16 · 10-datasets average | accuracy: 70.1 -> 72.7 | Helped |
| Apply Frolic to InMaP | InMaP, CLIP ViT-L/14 · 10-datasets average | accuracy: 76.9 -> 78.9 | Helped |
| Apply Frolic to InMaP | InMaP, CLIP ViT-B/16 · ImageNet and five shifts average | accuracy: 61.6 -> 63.1 | Helped |
| Apply Frolic to InMaP | InMaP, CLIP ViT-L/14 · ImageNet and five shifts average | accuracy: 72.4 -> 73.3 | Helped |
| Use downstream-data implicit prior estimation instead of Frolic bias correction (ablation) | CLIP ViT-B/16 · 10 datasets and ImageNet average | accuracy: 69.3 -> 70.9 | Hurt |
| Replace pseudo-labels with ground-truth labels for bias correction (ablation) | Frolic · 10 datasets and ImageNet average | accuracy: 70.9 -> 71.9 | Helped |
Surprising findings
- The Gaussian-only classifier f_g (68.8% on 10 datasets) already outperforms the bias-corrected CLIP baseline, and simple logit summation of CLIP and the Gaussian model (66.3%) is worse than the Gaussian model alone.
- Frolic combined with InMaP underperforms standalone Frolic on several ImageNet shift datasets (e.g., IN-V2 63.8 vs 64.7), which the authors attribute to InMaP hyperparameters tuned on ImageNet.
Limitations
- Assumes class-conditional features follow a mixture of Gaussians, which may not hold.
- Performance depends on the pre-trained model's knowledge matching the downstream task.
- Societal concerns about noisy or malicious content in web-crawled pre-training data.
Experimental setup
Datasets: ImageNet Caltech101 SUN397 DTD EuroSAT UCF101 OxfordPets Stanford Cars Flowers102 Food101 FGVC Aircraft ImageNetV2
Benchmarks: Pets Flowers Aircraft DTD EuroSAT Cars Food SUN Caltech UCF ImageNet ImageNetV2
Metrics: accuracy
Baselines: CLIP TPT PromptAlign SuS-X-SD TDA GPT4-Prompt CuPL-CLIP InMaP CoOp CoCoOp Tip-Adapter LFA
Builds on: CLIP InMaP Logit Adjustment TDE CuPL
Closest papers by meaning
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.89)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.89)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.89)
- ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization (AAAI 2024 · similarity 0.88)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.