Research map / Vision-language and multimodal / Multimodal LLMs
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies design choices for visually-conditioned language models (VLMs) under a unified 12-benchmark evaluation suite and an open training codebase, and distills insights into the Prismatic (Prism) VLM family at 7B/13B that outperforms InstructBLIP and LLaVA v1.5.
Problem
Design decisions for VLMs (image preprocessing, visual backbone, optimization, language model choice, training scale) are under-explored, and existing evaluations are inconsistent and subjective, making it hard to know which factors drive performance.
Approach
Holds the LLaVA v1.5 architecture fixed and changes one design axis at a time, measuring effects with objective standardized benchmarks and z-score aggregation with significance testing; combines the winning choices (single-stage training, fused DINOv2+SigLIP features, naive resize, higher resolution, more epochs and data) into Prism models.
Key results
- Single-stage training (dropping projector-only alignment stage) matches or improves aggregate performance (p=0.00558) while saving 20-25% training cost.
- Fusing DINOv2 and SigLIP features with naive resize at 384px gives the strongest visual representation, e.g. 79.18 VQAv2 and 61.06 RefCOCO-style gains of 5-10% on localization and challenge tasks over single backbones.
- Prism-DINOSigLIP 13B reaches 81.66 VQAv2 and 58.01 VizWiz versus LLaVA v1.5 13B at 78.13 VQAv2 and 56.66 VizWiz, outperforming InstructBLIP and LLaVA v1.5 across the suite.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| skip the projector pretraining stage and use single-stage training (ablation) | Prismatic VLM · 12-benchmark aggregate | normalized aggregate performance: p = 0.00558 | Helped |
| finetune the visual backbone instead of keeping it frozen (ablation) | Prismatic VLM · 12-benchmark suite, especially RefCOCO and OCID-Ref | normalized aggregate performance: p = 0.00381 | Hurt |
| use vision-language contrastive pretrained features (CLIP or SigLIP) rather than alternatives (ablation) | Prismatic VLM · 12-benchmark aggregate | normalized aggregate performance: p = 7.11e-8 | Helped |
| use naive resize rather than letterbox padding (ablation) | Prismatic VLM with CLIP or SigLIP · 12-benchmark aggregate | normalized aggregate performance: p = 0.0176 | Mixed |
| increase input resolution to 336px or 384px (ablation) | Prismatic VLM · 12-benchmark aggregate | normalized aggregate performance: p = 6.05e-4 | Helped |
| fuse DINOv2 features with SigLIP features (ablation) | Prismatic VLM · 12-benchmark aggregate | normalized aggregate performance: p = 0.00164 | Helped |
| fuse DINOv2 features with CLIP features (ablation) | Prismatic VLM · 12-benchmark aggregate | normalized aggregate performance: p = 0.37313 | No clear effect |
| use an instruct-tuned LM rather than a base LM (ablation) | Prismatic VLM · 12-benchmark aggregate | normalized aggregate performance: p = 0.34854 | No clear effect |
| replace Llama-2 with Mistral v1 (ablation) | Prismatic VLM · 12-benchmark aggregate | normalized aggregate performance: p = 0.03097 | Mixed |
| remove language-only ShareGPT co-training data (ablation) | Prismatic VLM · 12-benchmark aggregate | normalized aggregate performance: p = 0.13655 | No clear effect |
| include language-only safety co-training data (ablation) | Prismatic VLM, especially base-LM models · toxic and racist prompts | safeguards in outputs | Helped |
| train for two epochs instead of one (ablation) | Prismatic VLM · 12-benchmark aggregate | normalized aggregate performance: p = 0.00496 | Helped |
Surprising findings
- Finetuning the visual backbone substantially degrades performance, especially on localization tasks (e.g., RefCOCO drops to ~19-42 from ~64).
- Instruction-tuned LMs (Vicuña v1.5) give no statistically significant gain over base LMs (Llama-2), and base LMs produce less verbose, less hallucinated responses.
Limitations
- Architecture study limited to the patch-as-token design; downsampling architectures such as Perceiver not studied.
- Evaluation focuses on standardized objective benchmarks, not open-ended dialogue quality.
- Findings not verified at larger scales (70B+).
Experimental setup
Datasets: LLaVA v1.5 pretraining mixture (LAION, CC, SBU, LLaVA synthetic, VQAv2, GQA, OK-VQA, OCR-VQA, A-OKVQA, TextCaps, RefCOCO LVIS-Instruct-4V LRV-Instruct
Benchmarks: VizWiz VQAv2 GQA TextVQA RefCOCO RefCOCO+ RefCOCOg OCID-Ref VSR TallyQA POPE AI2D
Metrics: accuracy normalized z-score IoU-threshold accuracy one-sided Fisher t-test p-value
Baselines: LLaVA v1.5 7B LLaVA v1.5 13B InstructBLIP 7B InstructBLIP 13B
Builds on: LLaVA v1.5 CLIP DINOv2 SigLIP Vicuña v1.5
Closest papers by meaning
- Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.93)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.92)
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.92)
- VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.91)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.91)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.