Research map / Vision-language and multimodal / Multimodal LLMs
Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Visual Graph Arena, six graph-reasoning image tasks (isomorphism, Hamiltonian/shortest path, Hamiltonian/chordless cycle) trained on one layout and tested on another, showing vision models and MLLMs fail while humans succeed.
Problem
Current vision and multimodal models lack 'conceptualization': recognizing the same graph concept across different visual layouts. Existing graph datasets are not visual and existing visual graph benchmarks have ambiguous layouts and small size.
Approach
A layout-shift benchmark of six graph tasks rendered as images, training on one drawing style (Kamada-Kawai or random) and testing on another (planar or other), with human baselines and analysis of GPT-o1 anomalies (middle-score and easier-worse) interpreted as pseudo-intelligence.
Key results
- Humans scored 88.2-100% across VGA tasks, while all vision models (ViT, Swin-T, ConvNeXt, SigLIP, DINOv2) failed the hard isomorphism task.
- ConvNeXt reached 82.4% on shortest path under Kamada-Kawai layout, outperforming ViT and Swin-T by 17.7 and 16.7 points.
- GPT-o1 reached 55% on shortest path and 66-67% on Hamiltonian cycle, but much of the cycle gain is attributed to detecting leaf nodes; GPT-4o and Claude 3.5 Sonnet failed all tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ConvNeXt Base instead of ViT Base or Swin-T Base | Vision models trained on Visual Graph Arena · Shortest path (Kamada-Kawai) | accuracy: 82.4; surpassing ViT and Swin-T by 17.7 and 16.7 percentage points, respectively | Helped |
| Use ConvNeXt Base instead of ViT Base or Swin-T Base | Vision models trained on Visual Graph Arena · Visual Graph Arena tasks where models did not completely fail | accuracy | Helped |
| Use GPT-o1 rather than GPT-4o, Claude 3.5 Sonnet, Claude 3 Opus, or Google Gemini | Multimodal LLMs · Shortest path (random) | accuracy: 55% | Helped |
| Use GPT-o1 rather than GPT-4o, Claude 3.5 Sonnet, Claude 3 Opus, or Google Gemini | Multimodal LLMs · Hamiltonian cycle | accuracy: 67% | Helped |
| Exclude graphs with leaf nodes from evaluation | GPT-o1 on Hamiltonian cycle · Hamiltonian cycle | accuracy: 66.0 to near-random levels | Hurt |
| Use GPT-o1 instead of humans | Visual Graph Arena evaluation · Graph isomorphism | accuracy: 95.0 vs FAIL; 91.6 vs FAIL | Hurt |
Surprising findings
- GPT-o1 scored 29% on length-1 shortest paths (adjacent nodes) versus about 70% on length-2 paths, an 'easier-worse' anomaly the authors interpret as pseudo-intelligence.
- The Hamiltonian cycle success of GPT-o1 is largely explained by detecting leaf nodes, and falls to near-random when such cases are excluded.
Limitations
- Graphs limited to 8-9 nodes, and shortest paths only up to length 4.
- MLLM evaluation used only 100 random samples per task and only API-accessible models.
- Vision model results reported from a single fine-tuning run with best test-set checkpoint selection, which is a weak protocol.
Experimental setup
Datasets: Visual Graph Arena connected graphs with 8 and 9 nodes (Brendan McKay collection)
Benchmarks: Visual Graph Arena
Metrics: accuracy
Baselines: ViT Swin Transformer ConvNeXt SigLIP DINOv2 GPT-o1 GPT-4o Claude 3.5 Sonnet
Builds on: Kamada-Kawai layout NetworkX
Closest papers by meaning
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.92)
- VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.91)
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.91)
- Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning (ICLR 2026 · similarity 0.91)
- SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.91)
- Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models (ICML 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.