Research map / Graph learning / GNN architectures
Preference-driven Knowledge Distillation for Few-shot Node Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PKD, a preference-driven knowledge distillation framework that combines a GTA-prompt fine-tuned Llama-3.1-8B LLM with GNN teachers, using K-uncertainty node selection and RL-based (PPO) per-node teacher selection for few-shot node classification on text-attributed graphs.
Problem
Few-shot node classification on text-attributed graphs (TAGs) where GNNs need many labels, LLMs are costly to run at scale, and nodes have heterogeneous local topologies that a single message-passing mechanism cannot handle.
Approach
Select nodes for LLM annotation via K-uncertainty (KL disagreement among multiple teacher GNNs), annotate them with a graph-topology-aware fine-tuned LLM, then use an RL agent (the fine-tuned LLM with projector heads trained by simplified PPO) to pick the most suitable teacher GNN per node for distillation into a student GNN.
Key results
- PKD with Llama-3.1-8B (5 labeled nodes per class) reaches 91.14% on Cora and 86.31% on Texas, outperforming the listed baselines on most datasets.
- Ablation on Cora: adding GTA prompts, DNS and V_PR raises accuracy from 45.02% to 41.14%-relative stacked gains (full model reported at 91.14% overall setting); GTA, DNS and V_PR each contribute increments.
- PKD is slower than most baselines (7.314 s/epoch on Cora vs 0.006-4.100 s for baselines), reflecting the accuracy-efficiency trade-off.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply PKD with 5 labeled nodes per class | few-shot node classification · Cornell, Washington, Texas, Wisconsin, Amazon Ratings, Ogbn-Arxiv, Cora | accuracy (%): Cornell: 72.68 ± 1.0 -> 80.95 ± 1.1; Washington: 73.33 ± 0.8 -> 83.74 ± 0.4; Texas: 84.85 ± 1.1 -> 86.31 ± 0.5; Wisconsi | Helped |
| apply PKD with 5 labeled nodes per class | few-shot node classification · Wiki CS and Pubmed | accuracy (%): Wiki CS: 84.88 ± 0.2 -> 81.39 ± 0.4; Pubmed: 85.98 ± 0.6 -> 85.69 ± 0.3 | Hurt |
| add GTA prompts for graph topology understanding (ablation) | PKD Llama · Cora and Amazon Ratings | accuracy (%): Cora: ↑ 41.14; Amazon Ratings: ↑ 23.97 | Helped |
| add DNS-selected neighbors instead of adjacency-based neighbors (ablation) | PKD Llama · Cora and Amazon Ratings | accuracy (%): Cora: ↑ 30.99 -> ↑ 41.14; Amazon Ratings: ↑ 16.96 -> ↑ 23.97 | Helped |
| select expansion nodes by GNN preference rank instead of randomly (ablation) | PKD Llama · Cora and Amazon Ratings | accuracy (%): Cora: ↑ 30.99 -> ↑ 41.14; Amazon Ratings: ↑ 16.96 -> ↑ 23.97 | Helped |
| add negative cross-entropy loss to the student-accuracy reward (ablation) | PKD teacher selector · expanded training data | classification performance | Helped |
| add negative distillation loss to the reward with accuracy and negative cross-entropy (ablation) | PKD teacher selector · expanded training data | classification performance | Helped |
| increase the node annotation ratio from 10% to 48% (ablation) | PKD Llama #LN 5 · Amazon Ratings | accuracy (%): 50.27 ± 6.6 -> 66.79 ± 0.3 | Helped |
| increase the node annotation ratio from 10% to 48% (ablation) | PKD Llama #LN 5 · Cora | accuracy (%): 73.37 ± 5.1 -> 91.14 ± 0.3 | Helped |
| use LLM-generated labels rather than the same number of real labels | GCNII, IceBerg, and MSKD · Cora, Wiki CS, Washington, Wisconsin | accuracy (%): GCNII Cora: 77.74 -> 76.69; Wiki CS: 56.29 -> 51.18; Washington: 64.17 -> 70.83; Wisconsin: 60.94 -> 62.50; IceBerg Cora | Mixed |
| increase the number of labeled nodes per class | PKD Llama · all nine TAG datasets | accuracy (%): Cornell: 74.60 ± 2.1 -> 80.95 ± 1.1; Washington: 76.64 ± 0.9 -> 83.74 ± 0.4; Texas: 80.36 ± 1.3 -> 86.31 ± 0.5; Wisconsi | Helped |
| use PKD Llama with #LN 5 | Cora running-time benchmark · Cora | seconds per epoch: 7.314 | Hurt |
Surprising findings
- Using the same number of real labels as LLM-generated labels yields better accuracy for GCNII and IceBerg, indicating LLM annotations are noisier than ground truth.
- Simple Random and Voting teacher-selection strategies are defeated by PKD on all datasets, while the LLM-based selector beats them.
Limitations
- Designed only for text-attributed graphs; not applicable to general graphs.
- Training efficiency is limited, as PKD is slower than most baselines and LLM annotation adds cost.
- Sensitive to hyperparameters such as beta and eta.
Experimental setup
Datasets: Cornell Washington Texas Wisconsin Amazon Ratings Ogbn-Arxiv Wiki CS Pubmed Cora
Benchmarks: Cornell Washington Texas Wisconsin Amazon Ratings Ogbn-Arxiv Wiki CS Pubmed Cora
Metrics: node classification accuracy running time per epoch
Baselines: GCNII EGNN LLMGNN GAugLLM Self-training AGST IceBerg KDGA MSKD BGNN MTAAM FairGKD
Builds on: GCN GAT APPNP H2GCN PPO LoRA
Closest papers by meaning
- Large Language Model Meets Graph Neural Network in Knowledge Distillation (AAAI 2025 · similarity 0.91)
- Label-free Node Classification on Graphs with Large Language Models (LLMs) (ICLR 2024 · similarity 0.90)
- Leveraging Large Language Models for Node Generation in Few-Shot Learning on Text-Attributed Graphs (AAAI 2025 · similarity 0.90)
- LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings (NeurIPS 2024 · similarity 0.89)
- Beyond the Golden Teacher: Enhancing Graph Learning through LLM-GNN Co-teaching (NeurIPS 2026 · similarity 0.89)
- VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.