Research map / Vision-language and multimodal / Multimodal LLMs
GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GRAPHGPT-O, a DreamLLM-based multimodal LLM that encodes personalized-PageRank-sampled neighborhoods of multimodal attributed graphs (linearized or via a hierarchical Q-Former aligner) to generate interleaved text and image content for a target node.
Problem
Multimodal LLMs cannot natively use graph structure and node text/image semantics jointly to generate new node text and images on multimodal attributed graphs (MMAGs), facing graph size explosion, non-Euclidean structure, hierarchical modality dependency, and inference-order issues.
Approach
Samples neighbors via personalized PageRank, then encodes them either as linearized interleaved text/image tokens or via a hierarchical aligner (node-feature Q-Former followed by graph-structure Q-Former) that feeds graph tokens into DreamLLM, with sequential or parallel text/image inference.
Key results
- On ART500K, GRAPHGPT-O (Soft) reached KL-DV 0.4327 versus 0.4882 for DreamLLM, and on Amazon-Baby it reached 0.23 KL-DV versus 1.92 for DreamLLM.
- On Amazon-Beauty, GRAPHGPT-O (Soft) achieved perplexity 285.0 and CLIP-IT 27.38 versus DreamLLM's 514.7 and 20.84.
- Removing the graph-structure Q-Former raised ART500K KL-DV from 0.4327 to 9.1406, showing the graph encoder is essential for graph alignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add graph structure Q-Former (ablation) | GRAPHGPT-O · ART500K | CLIP-I2: 61.44 -> 72.64 | Helped |
| add graph structure Q-Former (ablation) | GRAPHGPT-O · ART500K | KL-DV: 9.1406 -> 0.4327 | Helped |
| add graph structure Q-Former (ablation) | GRAPHGPT-O · ART500K | Perplexity: 59.67 -> 59.80 | No clear effect |
| add node feature Q-Former (ablation) | GRAPHGPT-O · ART500K | Perplexity: 71.96 -> 59.80 | Helped |
| add node feature Q-Former (ablation) | GRAPHGPT-O · ART500K | KL-DV: 2.5050 -> 0.4327 | Helped |
| replace graph structure Q-Former with GNN (ablation) | GRAPHGPT-O · ART500K | CLIP-I2: 71.15 -> 65.53 | Hurt |
| replace graph structure Q-Former with GNN (ablation) | GRAPHGPT-O · ART500K | KL-DV: 0.49 -> 4.83 | Hurt |
| use GRAPHGPT-O (Soft) instead of DreamLLM | DreamLLM · ART500K | CLIP-I2: 71.15 -> 72.64 | Helped |
| use GRAPHGPT-O (Soft) instead of DreamLLM | DreamLLM · ART500K | Perplexity: 555.7 -> 59.8 | Helped |
| use GRAPHGPT-O (Soft) instead of DreamLLM | DreamLLM · Amazon-Beauty | CLIP-IT: 20.84 -> 27.38 | Helped |
| use GRAPHGPT-O (Soft) instead of DreamLLM | DreamLLM · Amazon-Baby | KL-DV: 1.92 -> 0.23 | Helped |
| replace DreamLLM with GRAPHGPT-O (Soft) | DreamLLM · Amazon-Beauty | Perplexity: 514.7 -> 285.0 | Hurt |
Surprising findings
- On ART500K, GRAPHGPT-O (Soft) has lower CLIP-I2 (72.64) than GRAPHGPT-O (Hard) (77.62), so the soft aligner is not uniformly better on image quality.
- Modality order and inference strategy (sequential vs parallel) do not consistently change performance, per the linearization study.
Limitations
- Treats the graph as homogeneous, ignoring multiple node and edge types.
- Evaluated on only 50 test nodes per dataset with a small number of training nodes (40,000 sampled).
Experimental setup
Datasets: ART500K Amazon-Baby Amazon-Beauty
Benchmarks: ART500K Amazon-Beauty Amazon-Baby
Metrics: CLIP-I2 perplexity CLIP-IT KL-DV
Baselines: DreamLLM Chameleon Janus Emu3 GNN-replaced hierarchical aligner
Builds on: DreamLLM BLIP-2 Q-Former personalized PageRank (APPNP-style sampling) Stable Diffusion CLIP
Closest papers by meaning
- Mario: Multimodal Graph Reasoning with Large Language Models (CVPR 2026 · similarity 0.91)
- GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning (CVPR 2026 · similarity 0.89)
- Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models (ICML 2026 · similarity 0.89)
- GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning (NeurIPS 2024 · similarity 0.88)
- DiGraphHal-Bench: Evaluating Multimodal Large Language Models on Complex Directed Graphs (CVPR 2026 · similarity 0.88)
- Multi-View Empowered Structural Graph Wordification for Language Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.