Research map / Vision-language and multimodal / Multimodal LLMs

GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning

NeurIPS 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GITA, which renders graphs as visual images and pairs them with textual descriptions and task queries for a VLM, and releases the GVLQA vision-language graph reasoning dataset with layout augmentation analysis.

Problem

LLM-based graph reasoning relies on textual graph encodings and ignores the vision modality, and there is no vision-language dataset for general graph reasoning.

Approach

Render graphs as visual graphs (with controllable layout, node shape, outline, and edge thickness variations) and feed them with textual descriptions and task queries into a VLM (LLaVA) fine-tuned with LoRA; layout augmentation is identified as the key factor for vision-based graph reasoning.

Key results

• On GVLQA-BASE fine-tuned, GITA-13B reaches 60.00% average accuracy versus 56.83% for Vicuna-13B; GITA-7B reaches 58.76% vs 54.15% for Vicuna-7B.
• Layout augmentation (GVLQA-AUGLY) raises vision-only GITA-7B average accuracy from 38.91% to 63.36%, with SP improving from 5.72% to 76.55%.
• On real-world link prediction and node classification, GITA-7B fine-tuned reaches 83.14% average vs 81.68% for Vicuna-7B; zero-shot GITA-7B 53.40% vs ~31% for LLMs.

Measured effects reported in the paper

Surprising findings

• Vision-only GITA-7B outperforms text-based LLMs on Cycle and BGM tasks, while text remains better for sequence-ordering tasks like TS, SP, and HP.
• Under fine-tuning, adding the vision modality degraded some tasks for 13B models, which the authors attribute to LoRA-only tuning limiting modality alignment.

Limitations

• Subgraph sampling for large graphs can lose critical structural information and gives imbalanced coverage.
• Fine-tuning restricted to LoRA on the text decoder, so vision-text alignment is limited, especially for 13B models.
• Inference is much slower than GNN baselines and visual layout confuses shortest-path reasoning without edge weights.

Experimental setup

Datasets: GVLQA-BASE NLGraph-full ca-GrQC ca-HepTh PolBlogs Cora CiteSeer

Benchmarks: GVLQA-BASE GVLQA augmentation subsets ca-GrQC ca-HepTh PolBlogs Cora CiteSeer

Metrics: accuracy (exact match) inference time

Baselines: GPT-4 Turbo LLaMA2-7B LLaMA2-13B Vicuna-7B Vicuna-13B GCN GraphSAGE

Builds on: LLaVA LoRA NLGraph Graphviz

Closest papers by meaning

• Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.90)
• VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context (ICML 2024 · similarity 0.90)
• DiGraphHal-Bench: Evaluating Multimodal Large Language Models on Complex Directed Graphs (CVPR 2026 · similarity 0.89)
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.89)
• $\texttt{G1}$: Teaching LLMs to Reason on Graphs with Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• GraphVis: Boosting LLMs with Visual Knowledge Graph Integration (NeurIPS 2024 · similarity 0.89)