Research map / Vision-language and multimodal / Multimodal LLMs
GraphVis: Boosting LLMs with Visual Knowledge Graph Integration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GraphVis, which renders retrieved ConceptNet subgraphs as images and uses curriculum LoRA fine-tuning of LLaVA-v1.6-Mistral-7B (graph comprehension first, then KG-enhanced QA) to boost commonsense QA and zero-shot VQA.
Problem
Existing KG-enhanced LLMs linearize KG triples into text or use GNN embeddings, underutilizing graph structure; it is unclear whether LVLMs can understand KG structure when given graph images.
Approach
Visualize 2-hop ConceptNet subgraphs as images with Graphviz and fine-tune an LVLM with a curriculum: first on synthetic graph-comprehension questions (node/edge counts, degrees, triple listing), then on textual QA paired with the visual subgraph; the synthetic graph images also transfer to zero-shot VQA benchmarks.
Key results
- GraphVis (LLaVA-v1.6-Mistral-7B, LoRA) reaches 82.8 on CSQA (+12.3 over base 70.5) and 85.5 on OBQA (+9.9 over base 75.6).
- On VQA, GraphVis improves ScienceQA image-accuracy by 4.32 points (68.86 to 73.18) and MMBench overall by 2.66 points (63.75 to 66.41).
- Curriculum fine-tuning beats joint fine-tuning by 4.51 points on CSQA (82.56 vs 78.05).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| integrate retrieved ConceptNet subgraphs as visual graphs and curriculum-fine-tune | LLaVA-v1.6-Mistral (7B) · CSQA | accuracy: 70.5 -> 82.8; +12.3 | Helped |
| integrate retrieved ConceptNet subgraphs as visual graphs and curriculum-fine-tune | LLaVA-v1.6-Mistral (7B) · OBQA | accuracy: 75.6 -> 85.5; +9.9 | Helped |
| train on synthetic graph images with curriculum-fine-tuned GraphVis | LLaVA-v1.6-Mistral (7B) · ScienceQA | accuracy: 68.86 -> 73.18; +4.32 | Helped |
| train on synthetic graph images with curriculum-fine-tuned GraphVis | LLaVA-v1.6-Mistral (7B) · MMBench | accuracy: 63.75 -> 66.41; +2.66 | Helped |
| add visual graph comprehension and KG-enhanced QA fine-tuning | LLaVA-v1.6-Mistral (7B) · POPE-ran | accuracy: 88.56 -> 89.73 | Helped |
| replace curriculum fine-tuning with joint training on comprehension and reasoning tasks (ablation) | GraphVis · CSQA | accuracy: 82.56 -> 78.05; 4.51% | Hurt |
| train graph tasks before OCR tasks in the fine-grained curriculum (ablation) | GraphVis · CSQA | accuracy: −1.56% | Hurt |
| train OCR tasks before graph tasks in the fine-grained curriculum (ablation) | GraphVis · CSQA | accuracy: +0.24% | Helped |
| add KAPING prompts to the base model (ablation) | LLaVA-v1.6-Mistral (7B) · CSQA | accuracy: 70.52 -> 67.65; −2.87% | Hurt |
| add KAPING prompts to jointly fine-tuned GraphVis (ablation) | GraphVis (Joint) · CSQA | accuracy: 78.05 -> 78.71; +0.66% | Helped |
| add KAPING prompts to curriculum-fine-tuned GraphVis (ablation) | GraphVis · CSQA | accuracy: 82.56 -> 81.74; −0.82% | Hurt |
| jointly train graph comprehension and QA tasks instead of curriculum fine-tuning (ablation) | GraphVis · ScienceQA | accuracy: 71.94 -> 73.18 | Hurt |
Surprising findings
- KAPING-style textual triple prompting degrades the base LVLM (70.52 to 67.65 on CSQA), and adding it to the curriculum-tuned GraphVis also slightly hurts (-0.82).
Limitations
- Experiments limited to 7B models with LoRA fine-tuning due to compute constraints.
- Only 2-hop subgraphs used to keep images interpretable; subgraph retrieval not improved.
- Only ConceptNet evaluated; generalization to other KGs untested, and no error bars reported.
Experimental setup
Datasets: ConceptNet CommonsenseQA OpenBookQA ScienceQA MMBench POPE
Benchmarks: CommonsenseQA OpenBookQA ScienceQA MMBench POPE
Metrics: accuracy F1
Baselines: QA-GNN GreaseLM KAPING KSL GNP GPT-4 FLAN-T5-xxlarge LLaVA-v1.6-Mistral-7B
Builds on: KAPING QA-GNN GreaseLM LLaVA-v1.6 LoRA
Closest papers by meaning
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.89)
- UKnow: A Unified Knowledge Protocol with Multimodal Knowledge Graph Datasets for Reasoning and Vision-Language Pre-Training (NeurIPS 2024 · similarity 0.89)
- GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning (NeurIPS 2024 · similarity 0.89)
- Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning (ICLR 2024 · similarity 0.89)
- Talk like a Graph: Encoding Graphs for Large Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.