Research map / Graph learning / Graph self-supervised learning
Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks text-space graph foundation models (GFMs) by curating 20+ text-attributed graph datasets and evaluating graph SSL, graph-prompt (OFA, Prodigy), LLM-projector (LLaGA), and link-prediction-specific models under four co-training/pre-training paradigms.
Problem
Text-space graph foundation models lack a comprehensive benchmark with unified settings and sufficiently diverse text-attributed graph datasets, making it hard to compare methods and understand when they transfer positively.
Approach
Define four GFM paradigms (co-training/pre-training x task-specific/cross-task), curate over 20 text-space datasets across citation, e-commerce, molecular, and other domains, and evaluate representative GFM building blocks under unified protocols, including simple SBERT-based baselines with no pretraining.
Key results
- Node-level co-training: GFMs such as OFA and GraphMAE generally underperform GCN trained from scratch on node classification (e.g., GCN ST Cit-Avg 76.39 vs OFA ST 77.09 on citation; co-training gains rare), with Observation that DGI benefits from more data.
- Link prediction: BUDDY co-trained on larger graphs reaches average 86.18 Hits@100 versus GCN-D 42.29, showing structural embeddings matter for positive transfer.
- Graph classification: OFA co-trained on molecular datasets surpasses single-dataset training on all listed datasets (e.g., PCBA 0.236 vs 0.202 single atom features).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Co-train OFA across all node-classification graphs | OFA · Node classification, all datasets | accuracy: 79.63 -> 78.24 | Hurt |
| Co-train LLaGA across all node-classification graphs | LLaGA · Node classification, all datasets | accuracy: 79.38 -> 78.01 | Hurt |
| Co-train DGI across all node-classification graphs | DGI · Node classification, all datasets | accuracy: 76.35 -> 76.96 | Helped |
| Co-train BUDDY on similar-domain large graphs | BUDDY · Link prediction, large graphs | Hits@100: 75.48 -> 86.18 | Helped |
| Co-train GCN on similar-domain large graphs | GCN · Link prediction, large graphs | Hits@100: 43.27 -> 42.29 | Hurt |
| Co-train OFA on molecular graph-classification datasets | OFA · Molecular graph classification | performance | Helped |
| Co-train OFA on link and graph tasks | OFA · Graph classification (Graph Avg) | performance: 72.24 -> 75.04 | Helped |
| Co-train OFA on node and graph tasks | OFA · Graph classification (Graph Avg) | performance: 72.24 -> 75.86 | Helped |
| Co-train OFA on node, link, and graph tasks | OFA · Graph classification (Graph Avg) | performance: 72.24 -> 76.43 | Helped |
| Co-train OneForAll across node and link tasks | OneForAll · Link prediction (Hits@100) | Hits@100: 74.05 -> 79.30 | Helped |
| Co-train LLaGA across node and link tasks | LLaGA · Link prediction (Hits@100) | Hits@100: 85.03 -> 84.10 | No clear effect |
| Replace the OFA GCN backbone with SGC (ablation) | OFA · Node classification co-training | performance | Helped |
Surprising findings
- Simple SBERT with cosine similarity zero-shot outperforms OFA by a large margin on Cora and History, suggesting cosine similarity over LLM embeddings is key to in-context learning.
- Co-training across node-level datasets does not benefit and can degrade performance; ignoring graph structure yields no scaling behavior and SGC with fixed aggregation beats learnable GCN aggregation.
Limitations
- Node-level co-training of GFMs shows no benefit over task-specific GCN, and link prediction GFMs struggle to scale to large graphs.
- OFA loss weights were set to 1 rather than tuned, and BGRL and GCC were omitted due to failure to find working hyperparameters.
- Graph-level datasets are dominated by molecular domain, so conclusions for other domains are not established.
Experimental setup
Datasets: Cora CiteSeer Pubmed Arxiv Arxiv-2023 History Child Computers Photo Sportsfit Products Amazon Ratings
Benchmarks: Cora CiteSeer Arxiv Arxiv-2023 History Child Photo Computers Sports Products Pubmed HIV
Metrics: accuracy Hits@100 ROC-AUC AUC-ROC AP
Baselines: GCN GraphMAE DGI BGRL GCC OFA Prodigy LLaGA BUDDY SEAL
Builds on: OneForAll Prodigy LLaGA GraphMAE BUDDY
Closest papers by meaning
- Position: Graph Foundation Models Are Already Here (ICML 2024 · similarity 0.90)
- GFT: Graph Foundation Model with Transferable Tree Vocabulary (NeurIPS 2024 · similarity 0.88)
- How Do Large Language Models Understand Graph Patterns? A Benchmark for Graph Pattern Comprehension (ICLR 2025 · similarity 0.87)
- Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models (NeurIPS 2024 · similarity 0.87)
- TEG-DB: A Comprehensive Dataset and Benchmark of Textual-Edge Graphs (NeurIPS 2024 · similarity 0.87)
- DTGB: A Comprehensive Benchmark for Dynamic Text-Attributed Graphs (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.