Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes descriptor soup and word soup, greedy selections of GPT descriptors or random word chains maximizing few-shot ImageNet training accuracy on frozen CLIP, plus a diversity loss for finetuning, to improve OOD few-shot accuracy with few parameters.
Problem
Zero-shot descriptor ensembling methods (GPT descriptors, WaffleCLIP) cannot use few-shot training data, and existing few-shot soft prompt tuning methods require many parameters, leaving a tradeoff between parameter efficiency and OOD accuracy.
Approach
Greedily select discrete descriptors or chains of words (from a 10k common-word list) that maximize few-shot training accuracy with frozen CLIP text embeddings, storing only discrete tokens; add a diversity loss (KD-style with per-descriptor soft predictions) to finetune from the word soup initialization.
Key results
- Word soup (m=8) with CE finetuning achieves 67.03 average XD accuracy versus 65.02 for CLIP zero-shot and 65.63 for GPT centroids (ViT-B/16)
- Word soup with 8 descriptors achieves 1/25 of CoOp's stored parameters on XD and 1/70 on DG while matching CoOp accuracy
- Stacking word soup on CoOp, MaPLe, CLIPood improves XD/DG average accuracy by roughly 0.8-1.0%, e.g. CoOp 70.0 to 70.2 XD and 59.25 to 60.25 INet-based DG
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add word-soup score-mean evaluation to Cross Entropy | Cross Entropy on CLIP ViT/B-16 · Cross-dataset evaluation target mean (10 datasets) | accuracy: 66.80 -> 67.43 | Helped |
| Add word-soup score-mean evaluation to Cross Entropy | Cross Entropy on CLIP ViT/B-16 · Domain generalization evaluation target mean (4 datasets) | accuracy: 60.39 -> 61.32 | Helped |
| Add word-soup score-mean evaluation to CoOp | CoOp on CLIP ViT/B-16 · Cross-dataset evaluation target mean (10 datasets) | accuracy: 66.52 -> 67.30 | Helped |
| Add word-soup score-mean evaluation to CoOp | CoOp on CLIP ViT/B-16 · Domain generalization evaluation target mean (4 datasets) | accuracy: 59.25 -> 60.25 | Helped |
| Add word-soup score-mean evaluation to MaPLe | MaPLe on CLIP ViT/B-16 · Domain generalization evaluation target mean (4 datasets) | accuracy: 59.32 -> 60.20 | Helped |
| Add word-soup score-mean evaluation to CLIPood | CLIPood on CLIP ViT/B-16 · Cross-dataset evaluation target mean (10 datasets) | accuracy: 66.50 -> 67.42 | Hurt |
| Add token offsets to descriptor soup | Descriptor soup on CLIP ViT/B-16 · Cross-dataset evaluation target mean (10 datasets) | accuracy: 66.18 -> 66.67 | Helped |
| Add token offsets to descriptor soup | Descriptor soup on CLIP ViT/B-16 · Domain generalization evaluation target mean (4 datasets) | accuracy: 59.25 -> 59.82 | Helped |
| Add diversity loss with λ = 0.25 (ablation) | CoOp initialized with word soup · Cross-dataset evaluation target mean (10 datasets) | accuracy: 66.21 -> 67.23 | Helped |
| Add diversity loss with λ = 0.25 (ablation) | CoOp initialized with word soup · Domain generalization evaluation target mean (4 datasets) | accuracy: 59.15 -> 60.20 | Helped |
| Add word soup directly to soft CoOp prompts (ablation) | CoOp on CLIP ViT/B-16 · Cross-dataset evaluation target mean (10 datasets) | accuracy: 70.0 -> 69.6 | Hurt |
| Add word soup directly to soft CoOp prompts (ablation) | CoOp on CLIP ViT/B-16 · Domain generalization evaluation target mean (4 datasets) | accuracy: 66.52 -> 66.59 | No clear effect |
Surprising findings
- Random word chains (WaffleCLIP-style) match or beat GPT descriptor centroids, suggesting gains come largely from ensembling rather than descriptor content
- Naive finetuning of word soup descriptors with cross-entropy often slightly decreases target accuracy because descriptors collapse together
Limitations
- Descriptor soup relies on an LLM (GPT) to generate candidate descriptors at training time
- Word soup loses the interpretability of descriptors
- Token offset trick for descriptor soup has significant computational cost
Experimental setup
Datasets: ImageNet-1K 16-shot GPT-generated descriptors for ImageNet classes (Menon and Vondrick) 10,000 most common English words
Benchmarks: ImageNet Caltech-101 Oxford-Pets Stanford-Cars Flowers-102 Food-101 FGVC-Aircraft SUN-397 DTD EuroSAT UCF-101 ImageNet-V2
Metrics: top-1 accuracy average accuracy over cross-dataset targets average accuracy over domain generalization targets
Baselines: CLIP zero-shot GPT centroids GPT score mean WaffleCLIP CoOp CoCoOp MaPLe CLIPood ProGrad KgCoOp ProDA VPT
Builds on: WaffleCLIP classification by description CoOp model soups knowledge distillation
Closest papers by meaning
- Zero-Shot Robustification of Zero-Shot Models (ICLR 2024 · similarity 0.87)
- No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance (NeurIPS 2024 · similarity 0.86)
- Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuning (CVPR 2025 · similarity 0.86)
- Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.86)
- Reproducible Vision-Language Models Meet Concepts Out of Pre-Training (CVPR 2025 · similarity 0.86)
- Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.