AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

IG Captioner: Information Gain Captioners are Strong Zero-shot Classifiers

Chenglin Yang, Siyuan Qiao, Yuan Cao, Yu Zhang, Tao Zhu, Alan Yuille, Jiahui Yu

ECCV 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Attribute-based zero-shot learning · 2 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
score classes with information gain log P(T|I) - alpha log P(T) and train with matching unimodal loss (IG captioner)generative captioner · ImageNet-1K zero-shot (ViT-B)top-1 accuracy (%): 44.6 -> 64.3Improved
IG captioner versus CLIP classifier on ImageNet zero-shotIG captioner · ImageNet-1K zero-shot (ViT-L)top-1 accuracy (%): 68.2 -> 69.8Improved
apply IG evaluation alone to a standard captioner using zero-intensity image for the prior (ablation)standard captioner · ImageNet-1K zero-shottop-1 accuracy: +9.0% (ViT-B), +7.9% (ViT-L)Improved
add IG training objective on top of IG evaluation (ablation)IG captioner · ImageNet-1K zero-shottop-1 accuracy: further +10.7% (ViT-B), +10.2% (ViT-L)Improved
raise alpha from 0.0 to 0.8 in the evaluation objective (ablation)IG captioner ViT-B · ImageNet-1K zero-shottop-1 accuracy (%): 45.4 -> 64.3Improved
raise alpha from 0.8 to 1.0 (full prior removal) (ablation)IG captioner ViT-B · ImageNet-1K zero-shottop-1 accuracy (%): 64.3 -> 62.9Hurt
use a separate unimodal language model plus captioner instead of the single IG captioner (ablation)IG captioner · ImageNet-1K zero-shot (ViT-L)top-1 accuracy (%): 69.8 -> 68.9Hurt
use equal loss weights (1.0/1.0) instead of 1.5/0.5 for multimodal and unimodal losses (ablation)IG captioner training · ImageNet-1K zero-shottop-1 accuracy (%): 64.3 -> 63.0Hurt
IG captioner versus CLIP on image->text retrievalIG captioner · MSCOCO 5K (ViT-B)image->text R@1 (%): 56.3 (CLIP) vs 47.4Hurt
IG captioner instead of standard captioner on text->image retrievalIG captioner · MSCOCO 5K (ViT-B)text->image R@1 (%): 45.1 -> 42.9Hurt
IG captioner instead of standard captioner after VQAv2 fine-tuningIG captioner · VQAv2 test-stdaccuracy (%): 74.5 -> 75.2Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy