Research map / Vision-language and multimodal / Multimodal LLMs

Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a controllable MLLM-based captioner that produces short, descriptive, dense, and AltText-fused synthetic captions, and studies how these interact with web AltText when pre-training CLIP, MM1-style MLLMs, and diffusion models.

Problem

Whether synthetic captions can fully replace noisy web-crawled AltText in pre-training multimodal foundation models, and which caption formats suit each model (CLIP, MLLMs, diffusion) remains poorly understood.

Approach

A two-stage fine-tuned MLLM captioner (MM1-3B, then human-aligned fine-tuning) generates controllable caption formats (SSC, DSC, DSC+, AFC); the paper finds mixing synthetic captions with AltText beats synthetic captions alone for CLIP, while MLLMs prefer detailed captions and diffusion models prefer DSC.

Key results

• For CLIP on VeCap-300M, SSC+AltText reaches 66.63% ImageNet zero-shot vs 65.70% for AltText alone, with over 10 points gain on COCO retrieval (T-I R@1 48.13 vs 36.98), while DSC alone drops ImageNet zero-shot to 27.30.
• For MM1 1.2B pre-training, a 66/33 synthetic-to-AltText mix gives the best 0-shot (37.02) and 8-shot (64.71) scores.
• On SFT benchmarks, DSC+ alone averages 58.7 vs 57.6 for AltText and 56.5 for LLaVA captions.

Measured effects reported in the paper

Surprising findings

• Training CLIP solely on higher-quality synthetic captions significantly hurts zero-shot ImageNet accuracy, and the more descriptive DSC performs worst for CLIP despite better alignment.
• For MLLMs, more detailed captions (DSC+) containing more hallucinations outperform less hallucinated LLaVA captions after SFT.

Limitations

• Core CLIP and diffusion experiments use an in-house setup and VeCap-300M; the captioner and its data are not released, limiting reproducibility.
• Hallucination remains in dense captions (DSC+ has the lowest CapScore, 75.74).
• Under CLIP-filtered DFN-2B, synthetic captions do not outperform AltText alone and cause about 4% ImageNet drop.

Experimental setup

Datasets: VeCap-300M DFN-2B COCO 2017 Stage-1-1M Stage-2-HA

Benchmarks: COCO retrieval Flickr30k retrieval ImageNet ImageNetV2 TextCore VQAv2 MMMU MME SEED POPE LLaVA-Bench GenEval

Metrics: recall@1 zero-shot accuracy linear probing accuracy few-shot accuracy FID CapScore ANA CHAIR

Baselines: LLaVA captions ShareGPT4V LaCLIP VeCLIP LLaVA-1.5 MiniGPT-4 InstructBLIP Shikra

Builds on: VeCLIP LaCLIP MM1 LLaVA Stable Diffusion 3

Closest papers by meaning

• CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.91)
• Image Captioning with Multi-Context Synthetic Data (AAAI 2024 · similarity 0.90)
• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.88)
• CaptionQA: Is Your Caption as Useful as the Image Itself? (CVPR 2026 · similarity 0.88)
• Improving Cross-Modal Alignment with Synthetic Pairs for Text-Only Image Captioning (AAAI 2024 · similarity 0.88)
• Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception (CVPR 2025 · similarity 0.87)