AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

K-LITE: Learning Transferable Visual Models with External Knowledge

Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang, Pengchuan Zhang, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, Kurt Keutzer, Trevor Darrell and 2 more

NeurIPS 2022 · Vision-language and multimodal › CLIP and zero-shot · approach: Open-vocabulary alignment from VLMs · 44 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add Wiktionary definition knowledge in pre-training and evaluationUniCL · ImageNet-1Kzero-shot accuracy: 28.16 -> 30.23Improved
Add Wiktionary definition knowledge in pre-training and evaluationUniCL · ICinW (20 datasets)zero-shot score: 27.15 -> 33.44Improved
Add Wiktionary definition knowledgeUniCL · ImageNet-1Kzero-shot accuracy: 83.74% to 85.80%Improved
Continue pre-training without knowledge rather than with knowledge (ablation)Florence · ImageNet-1Kzero-shot accuracy: 85.35% vs 85.80%Hurt
Use Wiktionary definitions instead of no external knowledgeGLIP · LVISbox mAP: 18.5 -> 21.3Improved
Use Wiktionary definitions instead of no external knowledgeGLIP · ODinW (13 datasets)box mAP: 27.5 -> 31.7Improved
Use WordNet hierarchy instead of Wiktionary definitions (ablation)K-LITE image classifier · ImageNet-1Kzero-shot accuracy: 30.23 -> 29.03Hurt
Use WordNet definitions instead of Wiktionary definitions (ablation)K-LITE image classifier · ICinW (20 datasets)zero-shot score: 33.44 -> 29.14Hurt
Use Wiktionary definitions instead of no knowledge (ablation)UniCL · ImageNet-1Kzero-shot accuracy: 28.16 -> 4.93 when knowledge is added to the baseline at test timeImproved
Remove knowledge at evaluation after knowledge-augmented training (ablation)K-LITE · ICinW (20 datasets)zero-shot score: 33.44 -> 29.03Hurt
Use two selective language branches instead of only the knowledge branch (ablation)K-LITE modularized model · ImageNet-1Kzero-shot accuracy: 28.40% to 28.90%Improved
Use the caption combine scheme rather than the default concat scheme (ablation)K-LITE · ImageNet-1Kzero-shot accuracy: 36.67 -> 42.36Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy