Research map / Graph learning / Graph self-supervised learning
Large Language Model Meets Graph Neural Network in Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LinguGKD, which instruction-tunes LLMs (Mistral-7B, Llama2-7B, Llama3-8B) as TAG teachers and distills layer-wise node features into GNN students via layer-adaptive contrastive loss, improving GCN/GAT/GraphSAGE/GIN on Cora, PubMed and Arxiv.
Problem
LLMs are strong on text-attributed graphs but too expensive to deploy, while GNNs are efficient but struggle with rich TAG semantics; transferring LLM knowledge into GNNs has been largely unexplored.
Approach
Instruction-tune LLMs with structural graph prompts to create teacher LLMs, then distill hop-wise (layer-adaptive) node features into student GNNs through an InfoNCE-based contrastive loss with learnable per-hop weights, jointly with the classification loss.
Key results
- Distilled GCN with LinguGraph-Llama3 teacher reaches 90.77% accuracy on Cora vs 86.53% vanilla GCN; average distillation gains 4.61% on Cora, 2.79% on PubMed, 3.85% on Arxiv.
- LinguGraph-Llama3 (8B) teacher achieves 91.51% (Cora), 95.59% (PubMed), 79.73% (Arxiv) accuracy, exceeding listed GNN and graph Transformer baselines.
- GraphSAGE distilled from Llama3 teacher reaches 91.70% on Cora, slightly above its teacher's 91.51%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill LinguGraph-Llama3 into GCN | GCN · Cora | accuracy: 86.53±0.92 -> 90.77±0.28 | Helped |
| Distill LinguGraph-Llama3 into GraphSAGE | GraphSAGE · Cora | accuracy | Helped |
| Apply graph instruction tuning to Llama3 (ablation) | Llama3 (8B) · Cora | accuracy: 14.92 -> 91.51 | Helped |
| Apply graph instruction tuning to Llama3 (ablation) | Llama3 (8B) · PubMed | accuracy: 30.17 -> 95.59 | Helped |
| Apply graph instruction tuning to Llama3 (ablation) | Llama3 (8B) · Arxiv | accuracy: 15.31 -> 79.73 | Helped |
| Use layer-adaptive, multi-hop contrastive distillation rather than aligning only the last-order features (ablation) | LinguGKD · Cora and PubMed | node classification performance | Helped |
| Increase neighbor order beyond 2 hops (ablation) | Vanilla GNNs · Cora | node classification performance | Hurt |
| Increase hidden feature dimension beyond 128 (ablation) | Vanilla GNNs · Cora | node classification performance | No clear effect |
| Increase hidden feature dimension (ablation) | Distilled GNNs · Cora | node classification performance | Helped |
Surprising findings
- A distilled GraphSAGE student (91.70%) slightly surpasses its LLM teacher (91.51%) on Cora.
- Pre-tuned LLMs without graph instruction tuning achieve only about 8-15% accuracy on Cora and 10-15% on Arxiv.
Limitations
- Student GNNs are trained on pre-extracted dataset embeddings (BoW/TF-IDF/Skip-gram), losing semantic information compared to raw text, limiting gains on PubMed.
- Teacher LLM instruction tuning on large prompt sets (up to 2M prompts on Arxiv) is computationally costly.
- Experiments are limited to node classification on three citation datasets, and the text attributes were reconstructed rather than original.
Experimental setup
Datasets: Cora PubMed Arxiv
Benchmarks: Cora PubMed Arxiv
Metrics: accuracy F1
Baselines: GCN GAT GraphSAGE GIN SGC BernNet FAGCN GCNII RevGAT ACM-GCN+ GraphTransformer Graphormer
Builds on: infoNCE LoRA GCN GAT GraphSAGE
Closest papers by meaning
- Preference-driven Knowledge Distillation for Few-shot Node Classification (NeurIPS 2025 · similarity 0.91)
- VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs (ICLR 2024 · similarity 0.89)
- A Unified Graph Language Model for Multi-Domain Multi-Task Graph Alignment Instruction Tuning (NeurIPS 2026 · similarity 0.89)
- LoReC: Rethinking Large Language Models for Graph Data Analysis (NeurIPS 2026 · similarity 0.89)
- Label-free Node Classification on Graphs with Large Language Models (LLMs) (ICLR 2024 · similarity 0.89)
- GOFA: A Generative One-For-All Model for Joint Graph Language Modeling (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.