Research map / Graph learning / GNN architectures
LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TEA-GLM, which pretrains a GraphSAGE encoder with instance- and feature-wise contrastive losses aligned to PCA components of Vicuna token embeddings, then trains a linear projector to map node representations into graph tokens for a frozen Vicuna-7B, enabling zero-shot cross-dataset and cross-task graph learning.
Problem
Zero-shot graph learning across datasets (differing label sets) and tasks (node classification vs. link prediction) without task-specific fine-tuning, where GNN representations transfer poorly to LLM predictors.
Approach
Align GNN node representations with the principal components of an LLM's token embedding space via a feature-wise contrastive loss during self-supervised pretraining, then map central-node representations to a fixed number of graph tokens with a linear projector and use a unified instruction that lists candidate answers, keeping the LLM frozen.
Key results
- On zero-shot node classification, TEA-GLM reaches 0.848 accuracy on Pubmed (vs. 0.793 for LLaGA) and 0.528 on History (vs. 0.363 for Vicuna-7B-v1.5).
- On cross-task link prediction, TEA-GLM attains AUC 0.689 on Pubmed and 0.657 on Arxiv, outperforming OFA, Vicuna, and LLaGA on most datasets.
- Removing graph token embeddings causes a significant performance drop on both node-level and edge-level tasks, per the ablation study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TEA-GLM instead of the strongest listed baseline | Zero-shot node classification · Pubmed, Cora, Children, History, Photo, and Sports | accuracy | Helped |
| add graph token embeddings to provide graph information to the LLM (ablation) | TEA-GLM · Cross-dataset and cross-task evaluations | performance | Helped |
| add feature-wise contrastive learning during GNN pretraining (ablation) | TEA-GLM · Unseen datasets | performance | Helped |
| add feature-wise contrastive learning during GNN pretraining (ablation) | TEA-GLM · Seen datasets (Arxiv and Computer), training task | performance | Hurt |
| add feature-wise contrastive learning during GNN pretraining (ablation) | TEA-GLM · Unseen link prediction tasks | performance | Helped |
| replace sum pooling with mean pooling (ablation) | TEA-GLM · Arxiv link prediction | AUC: 0.657 -> 0.659 | Helped |
| replace sum pooling with mean pooling (ablation) | TEA-GLM · Pubmed link prediction | AUC: 0.689 -> 0.690 | Helped |
| replace sum pooling with mean pooling (ablation) | TEA-GLM · Cora link prediction | AUC: 0.586 -> 0.588 | Helped |
| increase graph token embedding count (ablation) | TEA-GLM · Training datasets, node classification | Macro F1 | Helped |
| increase graph token embedding count (ablation) | TEA-GLM · Unseen datasets | Macro F1 | No clear effect |
| omit contrastive learning with principal components (ablation) | TEA-GLM · Supervised training datasets | accuracy | Helped |
| use too few principal components (ablation) | TEA-GLM · Node classification | performance | Hurt |
Surprising findings
- Vicuna-7B-SPT (soft prompt tuning) fails to transfer to e-commerce datasets, suggesting graph tokens rather than soft prompts carry the transferable information.
- TEA-GLM's accuracy on Sports (0.404) is not the best; LLaGA's macro F1 is higher on Sports, and TEA-GLM does not beat all baselines on every unseen task.
Limitations
- Graph-level tasks are not experimentally evaluated, only claimed to be applicable.
- Zero-shot cross-dataset evaluation restricted to a single domain pairing per setting (citation or e-commerce), and only node classification and link prediction tasks.
- Legality rate on some unseen datasets (Cora 92.6%) is below 100%, so LLM outputs can still fall outside candidate answers.
Experimental setup
Datasets: Arxiv Computer Pubmed Cora Children History Photo Sports
Benchmarks: Arxiv Pubmed Cora Children History Photo Sports Computer
Metrics: accuracy macro F1 AUC legality rate
Baselines: MLP GCN GraphSAGE GAT DGI GKD GLNN NodeFormer DIFFormer OFA Vicuna-7B-v1.5 Vicuna-7B-SPT
Builds on: GraphSAGE GraphGPT LLaGA GraphCL Deep Graph Infomax
Closest papers by meaning
- Leveraging Large Language Models for Node Generation in Few-Shot Learning on Text-Attributed Graphs (AAAI 2025 · similarity 0.91)
- A Unified Graph Language Model for Multi-Domain Multi-Task Graph Alignment Instruction Tuning (NeurIPS 2026 · similarity 0.89)
- Label-free Node Classification on Graphs with Large Language Models (LLMs) (ICLR 2024 · similarity 0.89)
- Preference-driven Knowledge Distillation for Few-shot Node Classification (NeurIPS 2025 · similarity 0.89)
- UniGTE: Unified Graph–Text Encoding for Zero-Shot Generalization across Graph Tasks and Domains (NeurIPS 2025 · similarity 0.89)
- GLBench: A Comprehensive Benchmark for Graph with Large Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.