Research map / Vision-language and multimodal / Multimodal LLMs
Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Prism decouples a VLM into a perception stage (VLM describes the image as text) and a reasoning stage (an LLM answers from that text), uses it to analyze VLM perception vs. reasoning on MMStar, and trains a 2B PrismCaptioner that rivals much larger VLMs.
Problem
Existing VLMs entangle seeing and reasoning, so their perception and reasoning abilities cannot be assessed independently, and capable general VLMs are costly to train and deploy.
Approach
Split visual question answering into a VLM-based textual description step and a text-only LLM reasoning step, so swapping either module isolates its contribution; a small captioner paired with a strong LLM then serves as a cheap VLM.
Key results
- PrismCaptioner-2B with ChatGPT reaches 43.3 on MMStar versus 34.5 for its end-to-end LLaVA-InternLM2-1.8B baseline, and is said to match VLMs about 10x larger such as LLaVA-InternLM2-20B and Yi-VL-34B.
- Decoupled GPT-4o (GPT-4o as both perception and reasoning module) reaches about 61% on MMStar, close to its end-to-end 61.6%.
- Ensemble of GPT-4v and GeminiPro-V perception outputs reaches 50.3 overall on MMStar, above either model alone (47.9 and 44.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace end-to-end LLaVA-InternLM2-1.8B with PrismCaptioner-2B and external ChatGPT reasoning | LLaVA-InternLM2-1.8B · MMStar | accuracy: 34.5 -> 43.3 | Helped |
| Replace end-to-end LLaVA-InternLM2-7B with PrismCaptioner-7B and external Llama3 reasoning | LLaVA-InternLM2-7B · MMStar | accuracy: 38.3 -> 45.9 | Helped |
| Use PrismCaptioner-7B with Llama3 rather than ChatGPT as reasoning module (ablation) | PrismCaptioner-7B · MMStar | accuracy: 43.7 -> 45.9 | Helped |
| Use PrismCaptioner-2B with Llama3 rather than ChatGPT as reasoning module (ablation) | PrismCaptioner-2B · MMStar | accuracy: 43.3 -> 42.0 | Hurt |
| Use Llama-3-70B-Instruct rather than GPT-3.5-Turbo-0125 as reasoning module (ablation) | GPT-4o perception module · MMStar | accuracy: 54.7 -> 59.3 | Helped |
| Use SigLip-SO400M rather than CLIP ViT-L/14 as vision backbone (ablation) | PrismCaptioner with InternLM2-7B · MMStar | accuracy: 41.7 -> 43.5 | Helped |
| Use SigLip-SO400M rather than InternViT-6B as vision backbone (ablation) | PrismCaptioner with InternLM2-7B · MMStar | accuracy: 42.7 -> 43.5 | Helped |
| Use Human 1 rather than GPT Synthesize 2 as generic instruction (ablation) | GPT-4o perception module · MMStar | accuracy: 48.1 -> 48.4 | Helped |
| Use Human 1 rather than GPT Synthesize 2 as generic instruction (ablation) | LLaVA-NeXT (Yi-34B) perception module · MMStar | accuracy: 40.1 -> 41.6 | Mixed |
| Increase language backbone from Vicuna-7B to Vicuna-13B (ablation) | LLaVA-v1.5 · MMStar perception evaluation | accuracy: 34.3 -> 33.6 | No clear effect |
| Increase language backbone from Vicuna-7B to Yi-34B (ablation) | LLaVA-NeXT · MMStar perception evaluation | accuracy: 39.3 -> 41.6 | Helped |
| Add query-specific instructions instead of generic instructions | GPT-4o perception module · MMStar | accuracy: 48.4 -> 54.5 | Helped |
Surprising findings
- Larger language models in LLaVA-v1.5 and LLaVA-NeXT yield only minor perception differences, suggesting perception is largely independent of LLM size.
- For large VLMs like GPT-4o and InternVL-Chat-v1.5, routing reasoning through an external ChatGPT may reduce overall performance.
Limitations
- Evaluation covers only a selected set of representative open-source and proprietary VLMs due to budget constraints.
- PrismCaptioner's training data is general-domain, so it has not been validated on domains like tables, diagrams, GUIs, or medical images.
- PrismCaptioner captions are less adaptive to query-specific instructions than GPT-4o's, limited by training data.
Experimental setup
Datasets: ALLaVA-Caption-4V Evol-Intruct-GPT4-Turbo-143K MMStar AI2D MMMU MathVista
Benchmarks: MMStar MMMU MathVista AI2D
Metrics: accuracy
Baselines: GPT-4o GPT-4v GeminiPro-V Qwen-VL-Max InternVL-Chat-v1.5 LLaVA-NeXT ShareCaptioner LLaVA-InternLM2-7B
Builds on: LLaVA QLoRA XTuner VLMEvalKit
Closest papers by meaning
- PRISM: Perception Reasoning Interleaved for Sequential Decision Making. (ICML 2026 · similarity 0.94)
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.93)
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.92)
- Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.91)
- Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs (ICLR 2026 · similarity 0.91)
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.