Research map / Graph learning / Graph self-supervised learning
GOFA: A Generative One-For-All Model for Joint Graph Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GOFA, a generative graph language model that interleaves randomly initialized token-level GNN layers into a frozen ICAE-style LLM compressor/decoder, pre-trained on graph completion, QA, structural and retrieval tasks, then instruction-tuned for zero-shot graph tasks.
Problem
Existing general graph models either serialize graphs into text for LLMs (weak structural understanding) or use GNNs with LLM assistants (no free-form generation), so no model is both structure-aware and generative for zero-shot graph tasks.
Approach
Interleave randomly initialized token-level GNN layers between LLM layers, with memory-token compression (ICAE-style) so each node's text is summarized into K tokens that message-pass per token index; a tanh gate initialized at zero keeps the initial model equal to the pretrained LLM; pre-train with graph-level next-word prediction, QA chains, structural (shortest path, common neighbors) and retri
Key results
- In zero-shot node classification with instruction tuning, GOFA-T reaches 71.17% on WikiCS (10-way) versus 63.63% for the best LLM baseline (Mistral-7B) and 43.45% for UniGraph.
- GOFA-F reaches 80.69% on FB15K237 (10-way) versus 62.48% for Mistral-7B, and 88.21 vs 89.54 Acc on supervised Cora link-level (GOFA best on several supervised tasks).
- Compared with an LLM given the same graph as text (LLM-N), GOFA-F improves ExplaGraphs accuracy from 74.13 to 79.49 while cutting per-sample inference time from 1.50 to 0.48 seconds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train GOFA with graph structural and semantic information | Mistral-7B · Cora, Product, WikiCS | perplexity: 30.12 -> 21.34 | Helped |
| Add GNN layers for graph structure modeling | Mistral-7B · Cora and Product | SPD RMSE: 1.254 -> 0.634 | Helped |
| Add GNN layers for graph structure modeling | Mistral-7B · Cora and Product | CN RMSE: 1.035 -> 0.326 | Helped |
| Add graph information through GOFA rather than textualizing the same graph input for an LLM | LLM-N · ExplaGraphs | accuracy: 74.13 -> 79.49 | Helped |
| Add graph information through GOFA rather than textualizing the same graph input for an LLM | LLM-N · Cora-Link | accuracy: 50.36 -> 85.10 | Helped |
| Add graph information through GOFA rather than textualizing the same graph input for an LLM | LLM-N · FB15K237 | accuracy: 51.25 -> 73.49 | Helped |
| Add Wikipedia datasets during pre-training (ablation) | GOFA · zero-shot downstream datasets | performance | Helped |
| Add the information retrieval pre-training task (ablation) | GOFA · knowledge graph tasks | performance | Helped |
| Increase the number of pre-training samples/tokens | GOFA · zero-shot downstream tasks | performance | Helped |
| Use GOFA instead of a textualized-graph LLM | LLM-N · WikiCS | accuracy | Helped |
| Use GOFA instead of Mistral-7B | Mistral-7B · Arxiv node count task | accuracy: 99.9 -> 86.3 | Hurt |
| Use GOFA instead of Mistral-7B | Mistral-7B · SceneGraphs QA | accuracy: 45.95 -> 34.06 | Hurt |
Surprising findings
- GOFA is only comparable to the Mistral-7B LLM on the SceneGraphs QA task, which the authors attribute to information-dense instruction data.
- On the additional structural tasks, GOFA drops to 86.3% on node count versus 99.9% for Mistral-7B, which the authors attribute to limited global structure pretraining.
Limitations
- Zero-shot evaluation covers unseen datasets with similar task formats rather than unseen task formats.
- The LLM compressor is kept frozen, so compression is not unified with graph data.
- Weaker on global structural tasks such as node count and on common-sense QA (SceneGraphs).
Experimental setup
Datasets: MAG240M PubMed ogbn-arxiv Wikikg90mv2 WikiGraph UltraChat200k Cora WikiCS Products FB15K237 ExplaGraphs SceneGraphs
Benchmarks: Cora WikiCS Products FB15K237 ExplaGraphs SceneGraphs BBBP HIV ogbn-arxiv
Metrics: accuracy perplexity RMSE AUROC
Baselines: LLaMA2-7B Mistral-7B OFA GraphGPT UniGraph ZeroG LLaGA GIMLET
Builds on: ICAE OFA TransformerConv LLaMA
Closest papers by meaning
- GRAVER: Generative Graph Vocabularies for Robust Graph Foundation Models Fine-tuning (NeurIPS 2025 · similarity 0.89)
- Large Language Model Meets Graph Neural Network in Knowledge Distillation (AAAI 2025 · similarity 0.88)
- GraphArena: Evaluating and Exploring Large Language Models on Graph Computation (ICLR 2025 · similarity 0.88)
- Multi-View Empowered Structural Graph Wordification for Language Models (AAAI 2025 · similarity 0.88)
- LLaGA: Large Language and Graph Assistant (ICML 2024 · similarity 0.88)
- GLBench: A Comprehensive Benchmark for Graph with Large Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.