AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

Improving Zero-Shot Generalization and Robustness of Multi-Modal Models

Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming-Hsuan Yang, Hartwig Adam, Laurent Itti, Balaji Lakshminarayanan, Jiaping Zhao

CVPR 2023 · Vision-language and multimodal › CLIP and zero-shot · approach: Attribute-based zero-shot learning · 27 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use WordNet parent-and-child prompt augmentation on uncertain examplesCLIP ViT-B/16 · ImageNet low-confidence settop-1 accuracy: 21.58% -> 38.71%Improved
Apply hierarchy augmentation only to the identified low-confidence subsetCLIP ViT-B/16 · ImageNet full settop-1 accuracy: 64.18% -> 67.78%Improved
Use WordNet parent-and-child prompt augmentationCLIP ViT-B/16 · ImageNet shifted datasetstop-1 accuracyImproved
Add hierarchy augmentationLiT ViT-B/32 · ImageNet full settop-1 accuracy: 68.26% -> 69.41%Improved
Combine hierarchy augmentation with prompt ensembling (ablation)CLIP ViT-B/16 · ImageNet-v2top-1 accuracy: 62.02% -> 62.00%No clear effect
Combine hierarchy augmentation with prompt ensembling (ablation)CLIP ViT-B/16 · ImageNet-Sketchtop-1 accuracy: 48.52% -> 48.26%Hurt
Combine hierarchy augmentation with prompt ensembling (ablation)CLIP ViT-B/16 · ImageNettop-1 accuracy: 68.48% -> 68.86%Improved
Vary confidence threshold from 0.47 to 0.70 (ablation)Hierarchy-CLIP · ImageNet full settop-1 accuracy: 68.72% to 68.86%No clear effect
Keep 30%, 20%, or 10% of WordNet nodes instead of all nodes (ablation)Hierarchy-CLIP · ImageNettop-1 accuracy: 68.52% -> 68.72%Improved
Use raw WordNet including rare academic parent terms (ablation)Hierarchy-CLIP · ImageNet zero-shot classificationtop-1 accuracyHurt
Use hierarchy augmentation across CLIP backbones (ablation)CLIP · ImageNet low-confidence settop-1 accuracy: +14.25% to +18.89%Improved
Use hierarchy augmentation across CLIP backbones (ablation)CLIP · ImageNet full settop-1 accuracy: +3.73% to +3.23%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy