Research map / Large language models / Language, safety and interpretability
When Do LLMs Help With Node Classification? A Comprehensive Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds LLMNodeBed, a PyG-based testbed of 14 text-attributed node classification datasets and 8 LLM-based methods, and runs over 2,700 models to derive design guidelines for when LLMs help node classification.
Problem
Practitioners lack clear design guidelines for when LLM-based node classification methods are worth their cost over classic GNNs and language models, since prior comparisons differ in backbones and codebases.
Approach
A standardized, fair benchmark that unifies backbones and codebases across LLM-as-Encoder, Explainer, Predictor, direct inference and graph foundation model categories, and analyzes learning paradigm, homophily, LLM type/size and prompt design as influencing factors.
Key results
- LLM-as-Encoder (LLMEmb/ENGINE) and Explainer (TAPE) methods reach best average accuracy in semi-supervised settings (e.g., GCN_LLMEmb 78.51 avg vs GCN 71.79), while supervised gains are marginal (TAPE 81.80 vs GCN 76.37 avg).
- In zero-shot, GPT-4o with w. Summary prompting reaches 64.99% average accuracy on four datasets, while ZeroG reaches 55.56% and LLaMA-8B w. Summary 53.27%.
- On heterophilic Texas, LLM encoders (Mistral-7B, Qwen-3B) outperform LM encoders by about 12% for GCN, SAGE and H2GCN in semi-supervised settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LLM-based methods instead of classic methods in semi-supervised node classification | Node classification · Semi-supervised datasets in Table 2 | accuracy: 71.79 -> 78.51 | Helped |
| use LLM-based methods instead of classic methods in supervised node classification | Node classification · Supervised datasets in Table 2 | accuracy: 78.70 -> 81.80 | Helped |
| use LLM-as-Explainer rather than classic methods | TAPE · Supervised, average across 10 datasets | accuracy: 78.70 -> 81.80 | Helped |
| add neighbor information to the GPT-4o prompt (ablation) | GPT-4o Direct Inference · Zero-shot, average across Cora, WikiCS, Instagram, Photo | accuracy: 61.30 -> 63.09 | Helped |
| add a neighbor summary to the GPT-4o prompt (ablation) | GPT-4o Direct Inference · Zero-shot, average across Cora, WikiCS, Instagram, Photo | accuracy: 61.30 -> 64.99 | Helped |
| use Chain-of-Thought prompting instead of Direct prompting (ablation) | GPT-4o Direct Inference · Zero-shot, average across Cora, WikiCS, Instagram, Photo | accuracy: 61.30 -> 62.28 | Helped |
| use Tree-of-Thought prompting instead of Direct prompting (ablation) | GPT-4o Direct Inference · Zero-shot, average across Cora, WikiCS, Instagram, Photo | accuracy: 61.30 -> 61.02 | No clear effect |
| use ReAct prompting instead of Direct prompting (ablation) | GPT-4o Direct Inference · Zero-shot, average across Cora, WikiCS, Instagram, Photo | accuracy: 61.30 -> 61.44 | No clear effect |
| replace Mistral-7B with GPT-4o as TAPE's explainer (ablation) | TAPE · Semi-supervised, average across 9 datasets | accuracy: 78.43 -> 79.17 | Helped |
| replace Mistral-7B with GPT-4o as TAPE's explainer (ablation) | TAPE · Supervised, average across 10 datasets | accuracy: 81.80 -> 81.97 | No clear effect |
| increase LLaGA's Qwen backbone size from 3B to 32B (ablation) | LLaGA · Supervised, average across 10 datasets | accuracy: 79.10 -> 79.79 | Helped |
| use Mistral-7B instead of Qwen-7B as LLaGA's backbone (ablation) | LLaGA · Supervised, average across 10 datasets | accuracy: 78.71 -> 81.69 | Helped |
Surprising findings
- LLM-as-Predictor methods show unstable semi-supervised performance with low Macro-F1 and imbalanced outputs.
- Advanced prompts (CoT, ToT, ReAct) give only minor gains over direct prompting for node classification, while neighbor-enriched prompts help more.
Limitations
- Limited coverage of heterophilic text-attributed datasets (only four, with Node2Vec shallow features).
- Zero-shot evaluation uses only a subset of datasets and 20% test nodes, and GFM comparisons are restricted to intra-domain transfer.
Experimental setup
Datasets: Cora Citeseer Pubmed ogbn-arXiv WikiCS Instagram Reddit Books Photo Computer Cornell Texas
Benchmarks: Cora Citeseer Pubmed arXiv WikiCS Instagram Reddit Books Photo Computer Cornell Texas
Metrics: accuracy macro-F1 training time inference time memory cost
Baselines: GCN GraphSAGE GAT MLP RoBERTa SenBERT GLEM ENGINE TAPE GraphGPT LLaGA ZeroG
Builds on: GLBench TAPE ENGINE LLaGA GraphGPT ZeroG GLEM H2GCN
Closest papers by meaning
- Label-free Node Classification on Graphs with Large Language Models (LLMs) (ICLR 2024 · similarity 0.92)
- How Do Large Language Models Understand Graph Patterns? A Benchmark for Graph Pattern Comprehension (ICLR 2025 · similarity 0.90)
- Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Data (ICML 2025 · similarity 0.89)
- Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models (NeurIPS 2024 · similarity 0.88)
- Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning (ICLR 2024 · similarity 0.88)
- Leveraging Large Language Models for Node Generation in Few-Shot Learning on Text-Attributed Graphs (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.