Research map / Vision-language and multimodal / Multimodal LLMs
Improved Baselines with Visual Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies LLaVA design choices and shows that an MLP connector, academic VQA data with response-format prompts, higher resolution, and a 13B LLM yield LLaVA-1.5, a strong, publicly-data-only baseline trainable in about a day on 8 A100s.
Problem
Unclear which design and data choices make large multimodal models strong on both short-answer academic VQA and open-ended visual conversation, and how to build an efficient, reproducible baseline.
Approach
Keep LLaVA's simple architecture but replace the linear projector with a two-layer MLP, add academic VQA data with explicit response-format prompts, scale input resolution to 336px and the LLM to 13B; add a grid-based high-resolution variant (LLaVA-1.5-HD) with a global context image.
Key results
- LLaVA-1.5-13B reaches 80.0 on VQAv2, 63.3 on GQA, 61.6 on SEED-Bench, and 36.1 on MM-Vet, using 558K pretraining and 665K finetuning samples.
- LLaVA-1.5-13B-HD (448px) improves to 81.8 VQAv2 and 39.4 MM-Vet, with LLaVA-1.5-7B already beating IDEFICS-80B on several benchmarks.
- Randomly subsampling the training mixture to 50% keeps over 98% of full-data performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace linear vision-language projection with a two-layer MLP connector (ablation) | LLaVA 7B (224px) · Academic and instruction-following benchmarks (GQA, MME, MM-Vet) | score: MME 1197.0 -> 1355.2 (vs +VQA-v2 row); MM-Vet 27.7 -> 27.8 region-level context | Helped |
| add response formatting prompt during finetuning (single word/phrase for VQA) (ablation) | LLaVA 7B with VQA-v2 · MME | perception score: 1197.0 -> 1323.8 | Helped |
| add VQAv2 to LLaVA training data (ablation) | LLaVA 7B (224px) · MME | perception score: 809.6 -> 1197.0 | Helped |
| add OKVQA and OCRVQA data (ablation) | LLaVA 7B (224px) · GQA / MME / MM-Vet | score: GQA 47.3 -> 50.0; MME 1355.2 -> 1377.6; MM-Vet 27.8 -> 29.6 | Helped |
| add region-level VQA data (Visual Genome, RefCOCO) (ablation) | LLaVA 7B (224px) · MM-Vet | score: 29.6 -> 30.8 | Helped |
| scale input resolution from 224 to 336 (CLIP-ViT-L-336px) (ablation) | LLaVA 7B · GQA / MME / MM-Vet | score: MM-Vet 30.8 -> 30.3; MME 1426.5 -> 1450; GQA 50.3 -> 51.4 | Mixed |
| add GQA training data (ablation) | LLaVA 7B (336px) · MME / MM-Vet | score: MME 1450 -> 1469.2; MM-Vet 30.3 -> 30.7; GQA score marked contaminated (62.0*) | Mixed |
| add ShareGPT conversation data (ablation) | LLaVA 7B (336px) · MME / MM-Vet | score: MME 1469.2 -> 1510.7; MM-Vet 30.7 -> 31.1 | Helped |
| scale LLM from Vicuna-7B to Vicuna-13B (ablation) | LLaVA-1.5 (336px) · MM-Vet / MME | score: MM-Vet 31.1 -> 36.1; MME 1510.7 -> 1531.3 | Helped |
| add global context (224px whole image concatenated) to high-res patch features (ablation) | LLaVA-1.5-HD 7B (448px) · GQA / MME / MM-Vet | score: GQA 62.9 -> 63.8; MME 1425.8 -> 1497.5; MM-Vet 31.9 -> 35.1 | Helped |
| scale to 448px with HD split-encode-merge and global context | LLaVA-1.5 13B · MM-Vet | score: 36.1 -> 39.4 | Helped |
| reduce LLaVA-1.5 training data by random subsampling to 50% | LLaVA-1.5 13B · MMBench, ScienceQA, POPE | score: performance does not decrease at 50%; retains >98% of full performance overall | No clear effect |
Surprising findings
- Downsampling the training data to 50% leaves MMBench, ScienceQA and POPE unchanged or slightly improved.
- Higher input resolution markedly reduces hallucination, suggesting hallucination from detailed training captions is partly a resolution-capacity issue.
Limitations
- Full image patch encoding makes training iterations long for high-resolution inputs.
- Cannot process multiple images because of lacking data and limited context length.
- Still hallucinates and has limited problem-solving in specialized domains such as medical.
Experimental setup
Datasets: LLaVA-Instruct 158K ShareGPT 40K VQAv2 GQA OKVQA OCRVQA A-OKVQA TextCaps RefCOCO Visual Genome CC-595K/LCS-558K pretraining
Benchmarks: VQAv2 GQA VizWiz ScienceQA-IMG TextVQA POPE MME MMBench MMBench-CN SEED-Bench LLaVA-Bench-in-the-Wild MM-Vet
Metrics: accuracy F1 GPT-4 score
Baselines: LLaVA InstructBLIP Qwen-VL-Chat BLIP-2 IDEFICS-80B Shikra PaLI-X-55B
Builds on: LLaVA InstructBLIP Vicuna CLIP
Closest papers by meaning
- LLaVA-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (ICLR 2025 · similarity 0.93)
- LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.91)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.90)
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.90)
- Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.90)
- LLaVA-Critic: Learning to Evaluate Multimodal Models (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.