Research map / Vision-language and multimodal / CLIP and zero-shot
VILA: On Pre-training for Visual Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic study of visual language pre-training design choices for auto-regressive VLMs, yielding the VILA model family that outperforms LLaVA-1.5 on 12 benchmarks.
Problem
Visual language model pre-training (aligning an LLM with visual inputs) has been understudied compared to visual instruction tuning; it is unclear which pre-training design choices (LLM freezing, data type, SFT blend) matter for downstream and text-only performance.
Approach
Controlled step-by-step ablations show that unfreezing the LLM during pre-training is needed for in-context learning, interleaved image-text corpora beat image-text pairs, and re-blending text-only instruction data into visual SFT recovers text ability while boosting VLM accuracy.
Key results
- Unfreezing the LLM during pre-training raises 4-shot VLM average accuracy from 37.5% (frozen, projector-only SFT) to 68.8%-70.9% range in ablations, with linear projector best at 70.9%.
- VILA-7B reaches 79.9 VQAv2, 57.8 VizWiz and 64.4 TextVQA, outperforming LLaVA-1.5 13B on VizWiz and TextVQA.
- Pre-training on interleaved MMC4 instead of image-text pairs (COYO) keeps MMLU degradation at about 5% versus 17.2% for COYO; joint SFT with text-only data recovers text-only accuracy to near Llama-2 SFT levels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune the LLM during visual-language pre-training instead of freezing it (ablation) | VILA pre-training · VLM ablation average (4 tasks), 4-shot | average accuracy/score: 44.5 -> 70.9 | Helped |
| use a linear projector instead of a Transformer projector (ablation) | VILA pre-training and SFT · VLM ablation average (4 tasks), 0-shot and 4-shot | average accuracy/score: 66.8 -> 68.7; 68.8 -> 70.9 | Helped |
| train only the projector during SFT, freezing the LLM (ablation) | VILA · visual-language tasks | performance | Hurt |
| pre-train on interleaved MMC4 rather than image-text pair data (ablation) | Llama-2 visual-language model · VLM accuracy average (4 tasks), 0-shot and 4-shot | accuracy/score: 51.1% -> 68.7%; 50.3% -> 70.9% | Helped |
| pre-train on image-text pairs alone using COYO (ablation) | Llama-2 visual-language model · MMLU | accuracy: 46.0% -> 28.8% (-17.2%) | Hurt |
| break interleaved MMC4 samples into image-text pairs (ablation) | Llama-2 visual-language model · VLM accuracy average (4 tasks), 4-shot | accuracy/score: 70.9% -> 44.5% | Hurt |
| blend MMC4 interleaved data with COYO image-text pairs (ablation) | Llama-2 visual-language model · VLM accuracy average (4 tasks), 0-shot and 4-shot | accuracy/score: 68.7% -> 69.0%; 70.9% -> 71.3% | Helped |
| blend text-only instruction data into visual instruction tuning (ablation) | VILA · VLM accuracy average (4 tasks), 0-shot and 4-shot | accuracy/score: 68.7% -> 71.0%; 70.9% -> 72.1% | Helped |
| blend text-only instruction data into visual instruction tuning (ablation) | VILA · MMLU | accuracy: 40.7% -> 51.4% | Helped |
| increase image resolution from 224 to 336 (ablation) | VILA with linear projector · TextVQA | accuracy: 41.6% -> 49.8% | Helped |
| downsample visual tokens at 336 resolution rather than use linear projection without token compression (ablation) | VILA · TextVQA | accuracy: 49.8% -> 45.6% | Hurt |
| fine-tune the LLM rather than add a visual expert while freezing it (ablation) | VILA visual-language pre-training · VLM accuracy average (4 tasks), 0-shot and 4-shot | accuracy/score: 67.0% -> 71.0%; 64.8% -> 72.1% | Helped |
Surprising findings
- Splitting interleaved MMC4 samples into image-text pairs (MMC4-pairs) gave lower VLM accuracy than COYO pairs and no in-context gain, showing interleaved structure matters beyond text distribution.
- A simple linear projector outperformed a Transformer-block projector when the LLM was unfrozen, which the authors attribute to forcing the LLM to learn more.
Limitations
- Pre-training corpus is about 50M images, far smaller than billion-scale corpora used by other VLMs.
- Smaller 7B model shows larger text-only degradation than the 13B model after pre-training.
- Token compression was not used in main results and is left to future work.
Experimental setup
Datasets: MMC4 COYO LAION FLAN LLaVA-1.5 SFT data ShareGPT4V MMLU BBH DROP
Benchmarks: VQAv2 GQA VizWiz ScienceQA-IMG TextVQA POPE MME MMBench MMBench-CN SEED-Bench LLaVA-Bench MM-Vet
Metrics: accuracy CIDEr 0-shot accuracy 4-shot accuracy
Baselines: LLaVA-1.5 InstructBLIP Qwen-VL-Chat BLIP-2 IDEFICS Shikra Visual Expert
Builds on: LLaVA-1.5 Flamingo MMC4 Llama-2 CLIP
Closest papers by meaning
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.92)
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.91)
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.91)
- Should VLMs be Pre-trained with Image Data? (ICLR 2025 · similarity 0.91)
- Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight (NeurIPS 2024 · similarity 0.90)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.