Research map / Vision-language and multimodal / CLIP and zero-shot
Scaling Up Vision-Language Pre-Training for Image Captioning
Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, Lijuan Wang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train the captioner on ALT200M rather than COCO alone | LEMON base · nocaps validation | CIDEr, in-domain and out-of-domain: +16.3 in-domain; +45.3 out-of-domain | Improved |
| Scale pre-training data from CC12M to ALT200M | LEMON base · nocaps validation overall | CIDEr: 100.4 -> 106.8 | Improved |
| Scale model size from base to huge with ALT200M pre-training | LEMON · nocaps validation overall | CIDEr: 106.8 -> 117.3 | Improved |
| Increase pre-training data while keeping model size small | LEMON scaling setup · COCO captioning | CIDEr | Mixed |
| Scale model size when pre-training data is only 3M | LEMON scaling setup · COCO captioning | CIDEr | No clear effect |
| Use larger models when pre-training data exceeds 40M | LEMON scaling setup · COCO captioning | CIDEr | Improved |
| Use s2s-MLM instead of LM (ablation) | Encoder captioner · Conceptual Captions val | CIDEr: 96.1 -> 104.4 | Improved |
| Use s2s-MLM instead of LM (ablation) | Encoder-decoder captioner · Conceptual Captions val | CIDEr: 94.9 -> 99.9 | Improved |
| Use s2s-MLM instead of LM (ablation) | Encoder-decoder captioner · COCO Karpathy test | CIDEr: 120.9 -> 120.4 | Hurt |
| Use encoder-decoder instead of encoder with LM (ablation) | Base-size captioner · Conceptual Captions val | CIDEr: 96.1 -> 94.9 | Hurt |
| Use encoder instead of encoder-decoder with s2s-MLM (ablation) | Base-size captioner · Conceptual Captions val | CIDEr: 99.9 -> 104.4 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- SimVLM huge — SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (ICLR 2022) · better on 3 of 3 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Unified Vision-Language Pre-Training for Image Captioning and VQA (AAAI 2020 · similarity 0.91)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.89)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.88)
- CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.88)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.88)
- Probabilistic Language-Image Pre-Training (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
