Research map / Large language models / Language, safety and interpretability
GLBench: A Comprehensive Benchmark for Graph with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GLBench, a benchmark that unifies data splits and implementations of 12 GraphLLM methods plus GNN/PLM baselines on 7 text-attributed graphs, evaluated in supervised and zero-shot settings with efficiency analysis.
Problem
Existing GraphLLM methods are evaluated with inconsistent datasets, splits, and protocols, so their relative performance, zero-shot transfer ability, and efficiency are unclear.
Approach
A unified benchmark that reimplements LLM-as-enhancer, LLM-as-predictor, and LLM-as-aligner GraphLLM methods under shared data processing, splits, and GNN/LLM backbones, adds a zero-shot protocol, and proposes a training-free baseline (Emb w/ NA) that combines frozen-LLM embeddings with neighborhood aggregation.
Key results
- In supervised settings GraphLLM methods achieve the best accuracy/F1 on 6 of 7 datasets; OFA beats GCN by 3.20% accuracy on Citeseer and LLaGA beats GCN by 4.00% accuracy on Reddit.
- In zero-shot settings, the training-free Emb w/ NA baseline outperforms OFA and GraphGPT across all datasets (e.g., 63.59% vs 23.11% on Cora for OFA).
- No clear model scaling law: OFA with LLaMA2-13B is best on Cora at 80.56%, but scaling GNN depth/width or LLM size shows no consistent gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an LLM-enhanced graph method rather than the best GNN baseline | Supervised node classification · Citeseer | accuracy: 69.84 -> 73.04; 3.20% | Helped |
| Align GNN and LLM representations and use the aligned GNN predictor | GLEM_GNN · Pubmed | accuracy: 79.10 -> 84.28; 5.18% | Helped |
| Use LLaGA as the predictor rather than the GNN baseline | Supervised node classification · Reddit | accuracy: 63.19 -> 67.19; 4.00% | Helped |
| Use LLM-as-predictor methods on graphs with limited training data | LLM-as-predictor methods · Cora, Citeseer, and Pubmed | accuracy | Hurt |
| Use LLM-as-enhancer methods | Supervised GraphLLM · Across datasets | accuracy and Macro-F1 | Helped |
| Use GLEM_GNN rather than GLEM_LLM or Patton | Aligned GraphLLM · 6 of 7 supervised datasets | accuracy | Helped |
| Increase GNN parameter count through layer width or depth (ablation) | OFA and GLEM · Cora | accuracy: 66.10 -> 72.97; 80.95, 79.01, 81.48, 81.24, 80.51 | No clear effect |
| Increase LLM backbone size (ablation) | OFA and GLEM · Cora | accuracy: 58.03; 80.56 | Mixed |
| Combine structural and semantic information for zero-shot transfer | OFA and ZeroG · All zero-shot datasets | accuracy | Helped |
| Use Emb w/ NA, combining text embeddings and neighborhood aggregation, instead of zero-shot OFA | Zero-shot transfer · Across all evaluated datasets | accuracy | Helped |
| Use LLMs with semantic information only, omitting graph structure | Zero-shot classification · Cora, Citeseer, Pubmed, WikiCS, and Instagram | accuracy | Mixed |
Surprising findings
- LLM-as-predictor methods underperform GNNs on citation datasets and often produce invalid or uncontrollable output labels.
- A training-free baseline combining frozen Sentence-BERT embeddings with neighborhood aggregation beats several zero-shot GraphLLM methods that require pre-training.
Limitations
- Only node classification is evaluated; edge-level and graph-level tasks are excluded.
- Only text-attributed graphs are considered; non-text graphs are not covered.
- Some LLM-as-predictor methods were run with default parameters due to long training time (e.g., InstructGLM ~20 hours per epoch on Arxiv), limiting tuning.
Experimental setup
Datasets: Cora Citeseer Pubmed Ogbn-arxiv WikiCS Reddit Instagram
Benchmarks: Cora Citeseer Pubmed Ogbn-arxiv WikiCS Reddit Instagram
Metrics: Accuracy Macro-F1 wall-clock training time peak GPU memory
Baselines: GCN GAT GraphSAGE BERT RoBERTa Sentence-BERT DGI GraphMAE GPT-4o
Builds on: TAPE OFA GLEM ENGINE ZeroG
Closest papers by meaning
- LoReC: Rethinking Large Language Models for Graph Data Analysis (NeurIPS 2026 · similarity 0.92)
- Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models (NeurIPS 2024 · similarity 0.90)
- Label-free Node Classification on Graphs with Large Language Models (LLMs) (ICLR 2024 · similarity 0.89)
- GraphArena: Evaluating and Exploring Large Language Models on Graph Computation (ICLR 2025 · similarity 0.89)
- How Do Large Language Models Understand Graph Patterns? A Benchmark for Graph Pattern Comprehension (ICLR 2025 · similarity 0.89)
- LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.