Research map / Vision-language and multimodal / Human and embodied modeling
Image Captioners Are Scalable Vision Learners Too
Michael Tschannen, Manoj Kumar, Andreas Peter Steiner, Xiaohua Zhai, Neil Houlsby, Lucas Beyer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace contrastive pretraining (CLIP*) by CapPa captioning pretraining | ViT-L/14 · COCO captioning with fresh decoder | CIDEr: 123.2 -> 125.8 | Improved |
| replace CLIP* by CapPa | ViT-L/14 · VQAv2 with fresh decoder | accuracy: 68.5 -> 70.9 | Improved |
| replace CLIP* (16k) by CapPa | ViT-B/16 · ImageNet-1k 10-shot-style linear adaptation with decoder | accuracy: 81.4 -> 81.3 | No clear effect |
| replace CLIP* by CapPa for full ImageNet finetuning | ViT-L/14 · ImageNet-1k finetuning | accuracy: 88.1 -> 87.7 | Hurt |
| add parallel prediction to Cap (CapPa) (ablation) | Cap · ImageNet 10-shot, 900M examples | accuracy: 49.7 -> 50.4 | Improved |
| add parallel prediction to Cap (CapPa) (ablation) | Cap · ImageNet-1k with decoder | accuracy: 80.2 -> 81.3 | Improved |
| randomly reverse captions in Cap (forward+backward) (ablation) | Cap · ImageNet 10-shot | accuracy: 49.7 -> 49.2 | No clear effect |
| use two 3-layer decoders instead of one 6-layer decoder in CapPa (ablation) | CapPa · ImageNet 10-shot | accuracy: 50.4 -> 49.5 | Hurt |
| use frozen T5 decoder instead of trained-from-scratch decoder in Cap (ablation) | Cap · ImageNet 10-shot | accuracy: 48.7 -> 42.8 | Hurt |
| unfreeze T5 decoder (ablation) | Cap with T5 decoder · ImageNet 10-shot | accuracy: 42.8 -> 48.6 | Improved |
| remove image input from Cap decoder (blind decoder) (ablation) | Cap · ARO VG Attribution | accuracy: 88.9 -> 83.7 | Hurt |
| apply captioning to ViT-B/16 instead of ResNet50 encoder (ablation) | Cap vs CLIP* · ImageNet zero-shot | relative accuracy | Improved |
Closest papers by meaning
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.90)
- FFF: Fixing Flawed Foundations in Contrastive Pre-Training Results in Very Strong Vision-Language Models (CVPR 2024 · similarity 0.90)
- CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.89)
- CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning (ICLR 2026 · similarity 0.89)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.89)
- LocCa: Visual Pretraining with Location-aware Captioners (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
