Research map / Vision-language and multimodal / CLIP and zero-shot
CapsFusion: Rethinking Image-Text Data at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CapsFusion, which uses ChatGPT (distilled into a finetuned LLaMA-2-13B) to fuse raw web captions with BLIP synthetic captions, producing the 120M-pair CapsFusion-120M dataset that improves LMM pretraining in scalability, sample efficiency, and world knowledge.
Problem
Models trained on large-scale synthetic captions from captioning models saturate early, degrade with more data, and lose world knowledge, while raw web captions are too noisy.
Approach
Use an LLM to consolidate raw web alt-text (knowledge) with synthetic captions (clean structure) into a single fused caption; ChatGPT outputs are distilled into an open LLaMA-2-13B so the fusion can run at 120M scale.
Key results
- CapsFusion captions improve CIDEr over synthetic captions by 18.8 on COCO and 18.3 on NoCaps at the 100M scale (115.7 vs 96.9 on COCO).
- CapsFusion reaches the best baseline's performance with 6-9M samples versus full baseline scale, claimed as an 11-16x sample-efficiency speedup.
- CapsFusion outperforms raw and synthetic captions on 7 of 9 SEED-Bench tasks; finetuned CapsFusion-LLaMA matches ChatGPT on 80 of 100 human-evaluated samples (equal or better).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse raw and synthetic captions with CAPSFUSION instead of using synthetic captions | LLaMA-2-7B LMM training · COCO | CIDEr: 96.9 -> 115.7 | Helped |
| Fuse raw and synthetic captions with CAPSFUSION instead of using synthetic captions | LLaMA-2-7B LMM training · NoCaps | CIDEr: 81.6 -> 99.9 | Helped |
| Fuse raw and synthetic captions with CAPSFUSION instead of using raw captions | LLaMA-2-7B LMM training · TextCaps | CIDEr: 59.6 -> 66.2 | Helped |
| Fuse raw and synthetic captions with CAPSFUSION instead of mixing them | LLaMA-2-7B LMM training · Flickr30K | CIDEr: 66.5 -> 73.0 | Helped |
| Train on synthetic rather than raw captions at 100M scale | LLaMA-2-7B LMM training · COCO | CIDEr: 85.5 -> 96.9 | Hurt |
| Use synthetic rather than raw captions at 100M scale | LLaMA-2-7B LMM training · SEED-Bench | accuracy: 48.7 -> 43.2 | Hurt |
| Increase synthetic-caption training scale from 10M to 100M | LLaMA-2-7B LMM training · COCO | CIDEr: 102.5 -> 96.9 | Hurt |
| Use CAPSFUSION captions instead of synthetic captions when tuning the LLM | LLaMA-2-7B · COCO | CIDEr: 55.5 -> 111.3 | Helped |
Surprising findings
- Synthetic captions outperform CapsFusion-style fusion at 10M scale but degrade at 100M, saturating around 30M pairs.
- Mixing raw and synthetic (or rewritten) captions does not give consistent gains in LMM training, unlike in CLIP training.
Limitations
- Fusion relies on ChatGPT-generated training data and an LLM that may still introduce hallucinations or errors.
- Evaluation of the LMM is limited to a frozen-LLM setup with only the bridging module trained, at 1 epoch per scale.
Experimental setup
Datasets: LAION-2B LAION-COCO COCO NoCaps TextCaps Flickr30K SEED-Bench MMLU
Benchmarks: COCO NoCaps TextCaps Flickr30K SEED-Bench MMLU
Metrics: CIDEr SPICE accuracy human preference win/tie counts
Baselines: Raw captions (LAION-2B) Synthetic captions (LAION-COCO) LaCLIP language rewrites Mixing raw and synthetic captions
Builds on: LaCLIP Emu BLIP LAION-COCO DataComp
Closest papers by meaning
- Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models (ICLR 2025 · similarity 0.91)
- MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning (CVPR 2026 · similarity 0.89)
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.89)
- Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception (CVPR 2025 · similarity 0.89)
- Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning (CVPR 2025 · similarity 0.89)
- CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.