Research map / Vision-language and multimodal / CLIP and zero-shot
Should VLMs be Pre-trained with Image Data?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies when to introduce image data in VLM training by resuming a 1.4B text LLM (DCLM-1B) at various pre-training checkpoints, mixing in image-caption data, and fine-tuning on LLaVA data, finding that mid-pretraining (~80%) image introduction with ~10-20% visual tokens works best.
Problem
Whether the standard two-step VLM pipeline (text-only LLM pre-training, then adding images) is optimal compared with introducing image data earlier in pre-training, and what image-text ratio, timing, and instruction-data choices matter for vision-language and text-only performance.
Approach
Instead of fully pre-training an LLM before adding images, interrupt text-only pre-training at intermediate checkpoints (e.g., 80%) and continue the existing learning-rate schedule with mixed image-caption and text data, then fine-tune; systematically sweep checkpoint timing, image ratio, instruction-data inclusion, and fine-tuning epochs.
Key results
- Introducing images at around 80% of text-only pre-training gives the best vision and text performance; the paper reports about a 2% average improvement over introducing images into a fully pre-trained model for a 1B model.
- At 7B scale, adding images at 50% of DCLM-7B pre-training improved VQAv2 from 74.69 to 76.39 and GQA from 60.6 to 61.08 relative to a no-image continued-training baseline.
- Adding instruction-tuning data during image pre-training harms vision-language performance, while 10-20% visual tokens is the best ratio at 1B scale.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Introduce image data during pre-training around 80% completion rather than after fully training the text model | 1B model · Average of 6 diverse tasks | average improvement: 2% | Helped |
| Introduce images during pre-training before the text model is fully cooled down | DCLM-7B · VQAv2 | accuracy: 74.69 -> 76.39 | Helped |
| Introduce images during pre-training before the text model is fully cooled down | DCLM-7B · VizWiz | accuracy: 11.69 -> 10.88 | Hurt |
| Add image data late during pre-training after the model has seen text-only data for most of pre-training | 1B model · Vision-language tasks | stable score | Helped |
| Add image data during pre-training from scratch instead of beginning with language-only training | 1B model · Vision-language tasks | stable score | Hurt |
| Add instruction-tuning image data to the image-text pre-training mix | 1B model · Vision-language tasks | VQA performance | Hurt |
| Add instruction-tuning image data to the image-text pre-training mix | 1B model · Text tasks | overall performance | Helped |
| Fine-tune with image instruction data for up to 4 epochs rather than stopping after one epoch | 1B model · Vision-language tasks | stable score | Helped |
| Fine-tune with image instruction data beyond 4 epochs | 1B model · Vision-language tasks | stable score | Hurt |
| Use DataComp-DR caption data instead of DataComp (ablation) | 1B model · Text Stable Score | stable score: 0.1168 -> 0.1503 | Helped |
| Use DataComp-DR caption data instead of DataComp (ablation) | 1B model · Vision Stable Score | stable score: 0.4603 -> 0.4607 | No clear effect |
| Use SigLIP + DINO instead of SigLIP (ablation) | 1B model · Vision Stable Score | stable score: 0.4607 -> 0.4696 | Helped |
Surprising findings
- Training from scratch with image data consistently hurt both vision and text performance at small scale, suggesting an early text-only phase is important.
- Including LLaVA instruction-tuning data in image pre-training degraded downstream VQA performance, even though it improved text-only tasks.
Limitations
- Experiments at 1B scale rely on only 28B image-text tokens per run, and training from scratch and higher token multipliers were not run at 1B scale.
- Only single image per sample, no interleaved data, and frozen image encoder; results may not transfer to other encoders or interleaved setups.
- Some benchmarks (VizWiz, OCID, etc.) were excluded or showed high variance, and the 1B model trails larger prior SOTA models.
Experimental setup
Datasets: DCLM-baseline StarCoder ProofPile DataCompDR-1B DataComp-1B CC12M LLaVA-1.5 instruction mix
Benchmarks: VQAv2 GQA TextVQA POPE RefCOCO OCID-Ref VizWiz AGIEval ARC-easy BigBench CS BigBench CC BoolQ
Metrics: accuracy stable score (accuracy minus random baseline, averaged)
Baselines: Fully pre-trained text model with re-warmup image training (Base 100%) Prismatic 7B PaliGemma 3B LLaMA 3.2 1B Qwen 2.5 1.5B
Builds on: DCLM Prismatic VLM LLaVA-1.5 DataComp-DR SigLIP
Closest papers by meaning
- VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.91)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.89)
- Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight (NeurIPS 2024 · similarity 0.89)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
- Why are Visually-Grounded Language Models Bad at Image Classification? (NeurIPS 2024 · similarity 0.88)
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.