Research map / Vision-language and multimodal / CLIP and zero-shot
IG Captioner: Information Gain Captioners are Strong Zero-shot Classifiers
Chenglin Yang, Siyuan Qiao, Yuan Cao, Yu Zhang, Tao Zhu, Alan Yuille, Jiahui Yu
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| score classes with information gain log P(T|I) - alpha log P(T) and train with matching unimodal loss (IG captioner) | generative captioner · ImageNet-1K zero-shot (ViT-B) | top-1 accuracy (%): 44.6 -> 64.3 | Improved |
| IG captioner versus CLIP classifier on ImageNet zero-shot | IG captioner · ImageNet-1K zero-shot (ViT-L) | top-1 accuracy (%): 68.2 -> 69.8 | Improved |
| apply IG evaluation alone to a standard captioner using zero-intensity image for the prior (ablation) | standard captioner · ImageNet-1K zero-shot | top-1 accuracy: +9.0% (ViT-B), +7.9% (ViT-L) | Improved |
| add IG training objective on top of IG evaluation (ablation) | IG captioner · ImageNet-1K zero-shot | top-1 accuracy: further +10.7% (ViT-B), +10.2% (ViT-L) | Improved |
| raise alpha from 0.0 to 0.8 in the evaluation objective (ablation) | IG captioner ViT-B · ImageNet-1K zero-shot | top-1 accuracy (%): 45.4 -> 64.3 | Improved |
| raise alpha from 0.8 to 1.0 (full prior removal) (ablation) | IG captioner ViT-B · ImageNet-1K zero-shot | top-1 accuracy (%): 64.3 -> 62.9 | Hurt |
| use a separate unimodal language model plus captioner instead of the single IG captioner (ablation) | IG captioner · ImageNet-1K zero-shot (ViT-L) | top-1 accuracy (%): 69.8 -> 68.9 | Hurt |
| use equal loss weights (1.0/1.0) instead of 1.5/0.5 for multimodal and unimodal losses (ablation) | IG captioner training · ImageNet-1K zero-shot | top-1 accuracy (%): 64.3 -> 63.0 | Hurt |
| IG captioner versus CLIP on image->text retrieval | IG captioner · MSCOCO 5K (ViT-B) | image->text R@1 (%): 56.3 (CLIP) vs 47.4 | Hurt |
| IG captioner instead of standard captioner on text->image retrieval | IG captioner · MSCOCO 5K (ViT-B) | text->image R@1 (%): 45.1 -> 42.9 | Hurt |
| IG captioner instead of standard captioner after VQAv2 fine-tuning | IG captioner · VQAv2 test-std | accuracy (%): 74.5 -> 75.2 | Improved |
Closest papers by meaning
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.91)
- Zero-Shot Image Captioning with Multi-type Entity Representations (AAAI 2025 · similarity 0.88)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.88)
- Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning (CVPR 2024 · similarity 0.88)
- MeaCap: Memory-Augmented Zero-shot Image Captioning (CVPR 2024 · similarity 0.88)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
