Research map / Vision-language and multimodal / Multimodal LLMs

DiGraphHal-Bench: Evaluating Multimodal Large Language Models on Complex Directed Graphs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DiGraphHal-Bench, a large-scale VQA benchmark of 2,796 directed graphs (Mermaid/Graphviz) evaluating MLLM hallucination and fine-grained structural, visual, semantic, and integrative reasoning, plus SFT specialist and curriculum experiments on Qwen2.5-VL-7B.

Problem

MLLMs hallucinate and reason poorly over complex directed graphs, where structural, visual, and semantic cues are intertwined; existing graph benchmarks are either synthetic with little semantics or real-world but small or LLM-judged, and lack a scale-quality balance.

Approach

A four-capability, twelve-task VQA benchmark over real-world procedural directed graphs, with a two-stage curation pipeline that pairs GPT-4o-generated semantic queries with deterministic algorithmic verification and traversal-based ground-truth answers.

Key results

• GPT-5 attains 81.79% accuracy on relative localization, while bounding-box accuracy at IoU 0.5 is below 10% for all models.
• Gemini 2.5 achieves only 48.20% F1 on Crossing Edge Pairs, showing near-universal failure on that edge layout task.
• SFT of Qwen2.5-VL-7B with a comprehensive-only specialist raises Non-Semantic element-centric F1 from 13.77 to 53.41.

Surprising findings

• Structural specialist SFT lets Qwen2.5-VL-7B surpass open-source baselines on several topological tasks, while the sequential curriculum model underperforms the baseline, indicating catastrophic forgetting.
• Edge attribute recognition collapses across models ('edge blindness'), while node attributes are recognized relatively well.

Limitations

• Closed-source models are evaluated only via API with one- or few-shot settings, limiting controlled comparison.
• Specialist models retain weaknesses on structurally-matched queries and thumbnail scale invariance.
• Semantic queries are generated by GPT-4o, which may introduce model-specific biases despite algorithmic verification.

Experimental setup

Datasets: FlowVQA BigDocs DiGraphHal-Bench training set

Benchmarks: DiGraphHal-Bench

Metrics: F1 accuracy Acc@0.5 Acc@0.25 existence accuracy

Baselines: GPT-4o GPT-5 o3 Gemini 2.5 Pro Claude Sonnet 4.5 GLM-4.5V Qwen2.5-VL-72B LLaVA-OneVision-72B Qwen2.5-VL-7B LLaVA-OneVision-7B InternVL3.5-8B

Builds on: FlowVQA BigDocs K-center-greedy selection

Closest papers by meaning

• Mario: Multimodal Graph Reasoning with Large Language Models (CVPR 2026 · similarity 0.90)
• Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.90)
• Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models (ICML 2026 · similarity 0.90)
• Debate on Graph: A Flexible and Reliable Reasoning Framework for Large Language Models (AAAI 2025 · similarity 0.89)
• Talk like a Graph: Encoding Graphs for Large Language Models (ICLR 2024 · similarity 0.89)
• GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning (NeurIPS 2024 · similarity 0.89)