Research map / Vision-language and multimodal / CLIP and zero-shot
Improving Zero-Shot Generalization and Robustness of Multi-Modal Models
Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming-Hsuan Yang, Hartwig Adam, Laurent Itti, Balaji Lakshminarayanan, Jiaping Zhao
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WordNet parent-and-child prompt augmentation on uncertain examples | CLIP ViT-B/16 · ImageNet low-confidence set | top-1 accuracy: 21.58% -> 38.71% | Improved |
| Apply hierarchy augmentation only to the identified low-confidence subset | CLIP ViT-B/16 · ImageNet full set | top-1 accuracy: 64.18% -> 67.78% | Improved |
| Use WordNet parent-and-child prompt augmentation | CLIP ViT-B/16 · ImageNet shifted datasets | top-1 accuracy | Improved |
| Add hierarchy augmentation | LiT ViT-B/32 · ImageNet full set | top-1 accuracy: 68.26% -> 69.41% | Improved |
| Combine hierarchy augmentation with prompt ensembling (ablation) | CLIP ViT-B/16 · ImageNet-v2 | top-1 accuracy: 62.02% -> 62.00% | No clear effect |
| Combine hierarchy augmentation with prompt ensembling (ablation) | CLIP ViT-B/16 · ImageNet-Sketch | top-1 accuracy: 48.52% -> 48.26% | Hurt |
| Combine hierarchy augmentation with prompt ensembling (ablation) | CLIP ViT-B/16 · ImageNet | top-1 accuracy: 68.48% -> 68.86% | Improved |
| Vary confidence threshold from 0.47 to 0.70 (ablation) | Hierarchy-CLIP · ImageNet full set | top-1 accuracy: 68.72% to 68.86% | No clear effect |
| Keep 30%, 20%, or 10% of WordNet nodes instead of all nodes (ablation) | Hierarchy-CLIP · ImageNet | top-1 accuracy: 68.52% -> 68.72% | Improved |
| Use raw WordNet including rare academic parent terms (ablation) | Hierarchy-CLIP · ImageNet zero-shot classification | top-1 accuracy | Hurt |
| Use hierarchy augmentation across CLIP backbones (ablation) | CLIP · ImageNet low-confidence set | top-1 accuracy: +14.25% to +18.89% | Improved |
| Use hierarchy augmentation across CLIP backbones (ablation) | CLIP · ImageNet full set | top-1 accuracy: +3.73% to +3.23% | Improved |
Closest papers by meaning
- No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance (NeurIPS 2024 · similarity 0.89)
- Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.88)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.88)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.88)
- Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learning (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
