AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

Improving multimodal datasets with image captioning

Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, Ludwig Schmidt

NeurIPS 2023 · Vision-language and multimodal › Human and embodied modeling · approach: Concept-guided image captioning · 22 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace discarded raw-caption examples with filtered BLIP2 captionsCLIP training on DataComp medium · DataComp medium (128M)ImageNet accuracy: 0.273* -> 0.317Improved
Mix filtered BLIP2 captions and filtered raw captionsCLIP training on DataComp medium · DataComp medium (128M)Average accuracy (38 tasks): 0.328* -> 0.376Improved
Use filtered BLIP2 captions instead of filtered raw captionsCLIP training on DataComp small · DataComp small (12.8M)ImageNet accuracy: 0.051* -> 0.080Improved
Add filtered BLIP2 captions with ImageNet-cluster image curationCLIP training on DataComp large · DataComp large (1.28B)ImageNet accuracy: 0.631* -> 0.643Improved
Replace discarded raw-caption examples with filtered BLIP2 captionsCLIP training on DataComp large · DataComp large (1.28B)Average accuracy (38 tasks): 0.529* -> 0.551Improved
Use BLIP2-generated captions rather than raw captionsCLIP training on DataComp medium · DataComp medium (128M)Average Flickr and MS-COCO retrieval: 19.7 -> 41.7Improved
Fine-tune the BLIP2 captioner on MS-COCO (ablation)CLIP training from BLIP2 captions · DataComp medium (128M)ImageNet accuracy: 0.281 -> 0.227Hurt
Fine-tune the BLIP2 captioner on MS-COCO (ablation)CLIP training from BLIP2 captions · DataComp medium (128M)Flickr retrieval: 0.507 -> 0.549Improved
Use a fine-tuned captioning model rather than its pre-trained version (ablation)CLIP training from captioner-generated text · DataComp medium (128M)ImageNet accuracy: 0.321 -> 0.252Hurt
Set BLIP2 sampling temperature to 0.75 rather than 1.0 (ablation)CLIP training from BLIP2 captions · DataComp medium (128M)ImageNet accuracy: 0.280 -> 0.281No clear effect
Set BLIP2 sampling temperature to 0.75 rather than 1.0 (ablation)CLIP training from BLIP2 captions · DataComp medium (128M)Average accuracy (38 tasks): 0.353 -> 0.357Improved
Use synthetic rather than raw captions for top-30%-filtered imagesCLIP training on DataComp medium · DataComp medium (128M)Average accuracy (38 tasks): +2.8%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy