Research map / Vision-language and multimodal / Multimodal LLMs
What matters when building vision-language models?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Controlled ablation study of VLM design choices (backbones, fully autoregressive vs cross-attention connectors, pooling, aspect-ratio handling, image splitting, training data) culminating in the open 8B Idefics2 model and its released data.
Problem
Many VLM design decisions (connector architecture, backbone choice, visual token pooling, image resolution handling, training data mixture and stability) are poorly justified and not controlled-compared, making it hard to tell which choices actually improve performance and efficiency.
Approach
Runs controlled ablations over VLM design axes and distills six findings, e.g., fully autoregressive architecture with a Perceiver resampler (64 visual tokens) plus LoRA on unimodal backbones outperforms cross-attention when backbones are trained, and image splitting at inference trades compute for OCR performance; applies these to build Idefics2.
Key results
- Idefics2-base with 64 visual tokens reaches 70.3 VQAv2 and 57.9 TextVQA in 8-shot evaluation, beating Flamingo-9B and MM1-7B on these tasks.
- Idefics2 (8B) reaches 43.5 MMMU val, 51.6 MathVista, 76.8 MMBench, outperforming LLaVA-NeXT-13B and DeepSeek-VL-7B on these benchmarks.
- Learned Perceiver pooling reduces visual tokens from 729 to 64 per image while improving average score by 8.5 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Llama-1-7B with Mistral-7B as the language backbone (ablation) | Cross-attention VLM · VQAv2 4-shot average | Avg. score: 62.5 -> 67.6 | Helped |
| use SigLIP-SO400M instead of CLIP-ViT-H as the vision encoder (ablation) | Cross-attention VLM · VQAv2 4-shot average | Avg. score: 57.4 -> 60.7 | Helped |
| use cross-attention rather than fully autoregressive fusion with frozen backbones (ablation) | VLM · VQAv2 4-shot average | Avg. score: 60.3 -> 66.7 | Helped |
| apply LoRA to the fully autoregressive model and train the backbones (ablation) | Fully autoregressive VLM · VQAv2 4-shot average | Avg. score: 51.8 -> 69.5 | Helped |
| add Perceiver learned pooling (ablation) | Fully autoregressive VLM · VQAv2 4-shot average | Avg. score: 51.8 -> 60.3 | Helped |
| increase Perceiver visual tokens from 64 to 128 (ablation) | Fully autoregressive VLM · VQAv2 4-shot average | Avg. score: 71.7 -> 71.2 | No clear effect |
| preserve the image's original aspect ratio and resolution (ablation) | SigLIP-based VLM · VQAv2 4-shot average | Avg. score: 73.1 -> 72.1 | No clear effect |
| replace alt-text with synthetic captions (ablation) | Image-text pair training · VQAv2 4-shot average | Avg. score: 49.8 -> 52.9 | Helped |
| add OCR training data at resolution 768 (ablation) | VLM · DocVQA | DocVQA score: 42.9 -> 49.9 | Helped |
| increase image resolution from 384 to 768 without OCR data (ablation) | VLM · DocVQA | DocVQA score: 22.6 -> 42.9 | Helped |
| use Perceiver rather than Linear Projection as the connector (ablation) | Fully autoregressive VLM · VQAv2 4-shot average | Avg. score: 44.5 -> 60.3 | Helped |
| increase the number of Perceiver resampler layers (ablation) | Fully autoregressive VLM · VQAv2 4-shot average | Avg. score: 69.3; 68.6; 69.0 | No clear effect |
Surprising findings
- Fully autoregressive architecture with unfrozen backbones outperforms cross-attention despite cross-attention having more trainable parameters, once LoRA is used.
- Unfreezing the vision and language backbones in the fully autoregressive architecture caused training divergence without LoRA.
- Increasing the number of visual tokens beyond 64 gave no gains, contrary to prior work.
Limitations
- Ablations run for only 6000 steps with a fixed setup, so conclusions may not transfer to longer training or other scales.
- Open-ended VQA metrics (e.g., VQAv2) penalize format differences, making some comparisons noisy.
- Red-teaming shows the model can produce biased judgments and help with some harmful requests such as CAPTCHA solving and phishing.
Experimental setup
Datasets: OBELICS PMD LAION COCO OCR-IDL PDFA Rendered Text The Cauldron LLaVA-Conv ShareGPT4V VQAv2 TextVQA OKVQA
Benchmarks: VQAv2 TextVQA OKVQA COCO MMMU MathVista MMBench DocVQA
Metrics: VQA accuracy CIDEr ANLS accuracy MMMU score
Baselines: OpenFlamingo Idefics1 Flamingo MM1 LLaVA-NeXT DeepSeek-VL MM1-Chat Gemini 1.5 Pro Claude 3 Sonnet Mini-Gemini-HD
Builds on: Flamingo IDEFICS LoRA DoRA NEFTune SigLIP
Closest papers by meaning
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.92)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.92)
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.92)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.92)
- Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.92)
- Can We Talk Models Into Seeing the World Differently? (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.