Research map / Vision-language and multimodal / CLIP and zero-shot
What If We Recaption Billions of Web Images with LLaMA-3?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Recaptions ~1.3B DataComp-1B web images with a LLaMA-3-8B-based LLaVA-1.5 captioner and shows the resulting Recap-DataComp-1B improves CLIP retrieval and text-to-image DiT alignment.
Problem
Web-crawled image-text pairs are noisy and have brief captions, and large-scale high-quality recaptioning has mostly been closed-source (relying on GPT-4V), limiting open research in vision-language pretraining and text-to-image generation.
Approach
Fine-tune an open LLaMA-3-8B-powered LLaVA-1.5 captioner (with extra HQ-Edit tuning) and use it to recaption the entire DataComp-1B dataset at billion scale, then train CLIP and DiT models on mixtures of original and recaptions.
Key results
- Recap-CLIP-B/16 with mixed ratio p=0.5 reaches ~5% retrieval boost over the p=1.0 baseline while ImageNet zero-shot drops slightly.
- On Urban-1K, Recap-CLIP-L/16 raises image-to-text retrieval from 69.8 to 89.0 and text-to-image from 64.6 to 91.8.
- DiT trained with p=0 improves COCO-Recap FID from 36.2 to 27.8 and CLIP score from 29.3 to 32.5 versus p=1.0; GPT-4V score rises by 1.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLaMA-3-8B as the LLaVA language decoder | LLaVA-1.5 · MMMU and MM-Vet | benchmark score: 33.6 -> 45.2; 33.9 -> 37.8 | Helped |
| Train CLIP with Recap-DataComp-1B captions | Recap-CLIP-B/16 · Urban-1K image-to-text retrieval | retrieval score: 53.2 -> 85.0 | Helped |
| Train CLIP with Recap-DataComp-1B captions | Recap-CLIP-B/16 · Urban-1K text-to-image retrieval | retrieval score: 50.9 -> 87.3 | Helped |
| Train CLIP with Recap-DataComp-1B captions | Recap-CLIP-B/16 · VG-Attribute | attribute score: 57.1 -> 66.4 | Helped |
| Replace original captions with recaptions in DiT training | DiT-BASE/4 · COCO-30K with COCO-Recap prompts | FID: 36.2 -> 27.8; -8.4 | Helped |
| Replace original captions with recaptions in DiT training | DiT-BASE/4 · COCO-30K with COCO-Recap prompts | CLIP Score: 29.3 -> 32.5; +3.1% | Helped |
| Replace original captions with recaptions in DiT training | DiT-BASE/4 · COCO-30K with COCO-Recap prompts | Recap-CLIP Score: 19.9 -> 28.3; +8.4% | Helped |
| Replace original captions with recaptions in DiT training | DiT-BASE/4 · COCO-30K with COCO-Recap prompts | GPT-4V Score: 1.40 -> 2.53; +1.1 | Helped |
| Increase the proportion of original captions from p=0.0 to p=0.8 (ablation) | Recap-CLIP-B/16 · ImageNet-1K validation | top-1 accuracy: 36.0 -> 69.2 | Helped |
| Use a mixture of original captions and recaptions instead of original captions only (ablation) | Recap-CLIP-B/16 · COCO image-to-text retrieval | R@1: 57.3 -> 62.7 | Helped |
| Use a mixture of original captions and recaptions instead of original captions only (ablation) | Recap-CLIP-B/16 · ImageNet-1K validation | top-1 accuracy: 69.7 -> 69.2 | Hurt |
| Replace the base text encoder with the large text encoder (ablation) | Recap-CLIP-B/16 · COCO image-to-text retrieval | R@1: 61.5 -> 62.9; +1.4% | Helped |
Surprising findings
- Recaptions slightly hurt ImageNet zero-shot classification even while improving retrieval.
- Using raw COCO captions for evaluation shows no significant FID improvement for recaption-trained DiT, the gains appear only with recaptioned evaluation prompts.
Limitations
- Recap-CLIP and DiT trained on recaptions degrade ImageNet zero-shot classification and need mixing with original captions.
- Evaluation of caption quality relies on GPT-4V ratings and CLIP scores, and standard CLIP scores show nearly no advantage for recaptions.
- Compute and hardware details are not reported.
Experimental setup
Datasets: DataComp-1B LAION CC SBU LLaVA-1.5 instruction data HQ-Edit ImageNet-1K MSCOCO Flickr30K Urban1K VG-Attribution
Benchmarks: ImageNet-1K MSCOCO 2014 Flickr30K Urban-1K VG-Attribution MMMU MM-Vet COCO-30K FID
Metrics: top-1 zero-shot accuracy recall@1 FID CLIP score Recap-CLIP score GPT-4V rating LongCLIP score
Baselines: OpenAI-CLIP-B/16 OpenCLIP-B/16 NegCLIP LLaVA-1.5-7B LLaVA-1.5-13B GPT-4V
Builds on: LLaVA-1.5 CLIPA DiT LaCLIP DALL-E 3
Closest papers by meaning
- LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding (CVPR 2026 · similarity 0.87)
- A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.86)
- LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation (AAAI 2025 · similarity 0.86)
- Reconstruction Alignment Improves Unified Multimodal Models (ICLR 2026 · similarity 0.86)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.86)
- Sieve: Multimodal Dataset Pruning using Image Captioning Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.