Research map / Vision-language and multimodal / Multimodal LLMs
Do Vision-Language Models Really Understand Visual Language?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a test suite of synthetic and real diagrams to test whether LVLMs understand entities and relations, and finds LVLMs handle entities well but fail on relations in synthetic diagrams, with apparent real-diagram success attributed to background-knowledge shortcuts.
Problem
Recent LVLMs appear to solve diagram reasoning tasks, but it is unclear whether they truly understand the entities and relations encoded in diagrams rather than relying on other cues.
Approach
A controlled evaluation suite decomposing diagram understanding into entity/relation recognition (name recognition) and counting-based reasoning, crossed with knowledge-free vs knowledge-required questions, over synthetic diagrams and 1,001 annotated real diagrams. Analyses show knowledge in diagrams acts as a recognition shortcut.
Key results
- On synthetic diagrams, average accuracy on implicit-relation counting questions is about 40.9%, and explicit-relation counting about 52.6%, while entity counting averages 81.8% (text) and 71.4% (visual).
- On real diagrams, models recognize relations well: average name-recognition accuracy on knowledge-required relation questions is 68.7%, and three large models average 81.1%.
- Grounding synthetic diagrams with semantic knowledge raises relation-recognition accuracy by about 12 points on average (58.9% to 70.9%), while counting accuracy changes little (52.6% to 53.3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add semantic knowledge to relations in synthetic diagrams | LVLMs on synthetic diagrams · Explicit relation recognition | accuracy: 58.9 -> 70.9; +12.0 | Helped |
| Add semantic knowledge to relations in synthetic diagrams | LVLMs on synthetic diagrams · Explicit relation counting | accuracy: 52.6 -> 53.3; +0.7 | No clear effect |
| Ask relation-recognition questions that require background knowledge | LVLMs on real diagrams · Relation recognition | accuracy: 60.0 -> 68.7; +8.7 | Helped |
| Ask relation-counting questions that require background knowledge | LVLMs on real diagrams · Relation counting | accuracy: 52.0 -> 54.5; +2.5 | No clear effect |
| Ask entity-recognition questions requiring background knowledge rather than knowledge-free questions | LVLMs on real diagrams · Entity recognition | accuracy: 79.7 -> 82.6; +2.9 | No clear effect |
| Ask entity-counting questions requiring background knowledge rather than knowledge-free questions | LVLMs on real diagrams · Entity counting | accuracy: 62.2 -> 61.8; -0.4 | No clear effect |
| Use in-context learning examples | LVLMs on synthetic diagrams · Relation recognition and reasoning | accuracy | No clear effect |
| Increase diagram complexity by increasing the number of entities | LVLMs on real diagrams · Relation questions | accuracy | Hurt |
| Remove all depicted relations from a real diagram | GPT-4o · Food-web relation recognition and food-chain counting | accuracy | No clear effect |
| Replace depicted relations with random relations | GPT-4o · Food-web relation recognition and food-chain counting | accuracy | No clear effect |
Surprising findings
- GPT-4V's counting of implicit (relative-position) relations is near random guessing (34.4%) on synthetic diagrams.
- Models recognize relations better in real diagrams than in simple synthetic diagrams, contrary to the authors' initial intuition.
Limitations
- Only diagrams representable as entity-relation graphs are considered.
- No method is proposed to fix the observed limitations.
- Uses a simple prompt and a basic answer parser, which may underestimate performance.
Experimental setup
Datasets: AI2D FoodWeb
Benchmarks: VLQA
Metrics: accuracy
Baselines: LLaVA-OneVision-7B Molmo Qwen2-VL-72B GPT-4V GPT-4o Gemini 1.5 Pro
Builds on: Chain-of-Thought prompting AI2D FoodWeb Word2Vec
Closest papers by meaning
- Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs (ICLR 2026 · similarity 0.92)
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Can Large Language Models Understand Symbolic Graphics Programs? (ICLR 2025 · similarity 0.88)
- Primitive Vision: Improving Diagram Understanding in MLLMs (ICML 2025 · similarity 0.88)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.88)
- Caption This, Reason That: VLMs Caught in the Middle (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.