Research map / Vision-language and multimodal / Human and embodied modeling
Improving multimodal datasets with image captioning
Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, Ludwig Schmidt
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace discarded raw-caption examples with filtered BLIP2 captions | CLIP training on DataComp medium · DataComp medium (128M) | ImageNet accuracy: 0.273* -> 0.317 | Improved |
| Mix filtered BLIP2 captions and filtered raw captions | CLIP training on DataComp medium · DataComp medium (128M) | Average accuracy (38 tasks): 0.328* -> 0.376 | Improved |
| Use filtered BLIP2 captions instead of filtered raw captions | CLIP training on DataComp small · DataComp small (12.8M) | ImageNet accuracy: 0.051* -> 0.080 | Improved |
| Add filtered BLIP2 captions with ImageNet-cluster image curation | CLIP training on DataComp large · DataComp large (1.28B) | ImageNet accuracy: 0.631* -> 0.643 | Improved |
| Replace discarded raw-caption examples with filtered BLIP2 captions | CLIP training on DataComp large · DataComp large (1.28B) | Average accuracy (38 tasks): 0.529* -> 0.551 | Improved |
| Use BLIP2-generated captions rather than raw captions | CLIP training on DataComp medium · DataComp medium (128M) | Average Flickr and MS-COCO retrieval: 19.7 -> 41.7 | Improved |
| Fine-tune the BLIP2 captioner on MS-COCO (ablation) | CLIP training from BLIP2 captions · DataComp medium (128M) | ImageNet accuracy: 0.281 -> 0.227 | Hurt |
| Fine-tune the BLIP2 captioner on MS-COCO (ablation) | CLIP training from BLIP2 captions · DataComp medium (128M) | Flickr retrieval: 0.507 -> 0.549 | Improved |
| Use a fine-tuned captioning model rather than its pre-trained version (ablation) | CLIP training from captioner-generated text · DataComp medium (128M) | ImageNet accuracy: 0.321 -> 0.252 | Hurt |
| Set BLIP2 sampling temperature to 0.75 rather than 1.0 (ablation) | CLIP training from BLIP2 captions · DataComp medium (128M) | ImageNet accuracy: 0.280 -> 0.281 | No clear effect |
| Set BLIP2 sampling temperature to 0.75 rather than 1.0 (ablation) | CLIP training from BLIP2 captions · DataComp medium (128M) | Average accuracy (38 tasks): 0.353 -> 0.357 | Improved |
| Use synthetic rather than raw captions for top-30%-filtered images | CLIP training on DataComp medium · DataComp medium (128M) | Average accuracy (38 tasks): +2.8% | Improved |
Closest papers by meaning
- Image Captioning with Multi-Context Synthetic Data (AAAI 2024 · similarity 0.92)
- Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models (ICLR 2025 · similarity 0.91)
- CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.91)
- Sieve: Multimodal Dataset Pruning using Image Captioning Models (CVPR 2024 · similarity 0.90)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.89)
- Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
