Research map / Label-efficient and robust learning / Knowledge distillation
Self-Training Based Few-Shot Node Classification by Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KD-FSNC, a self-training few-shot node classification method that distills GCN teacher knowledge into a student via local/global representation distillation and pseudo-label distillation.
Problem
Self-training few-shot node classification under-uses base-set information (only first GCN layer transferred) and is harmed by incorrect or over-confident pseudo-labels.
Approach
Pre-train a GCN teacher on the base set and train a randomly initialized student GCN on the novel set using (i) local representation distillation preserving neighbor embedding structure, (ii) global representation distillation matching teacher and student embeddings at all layers, and (iii) pseudo-label distillation that uses teacher class-prototype soft labels to supervise student soft prediction
Key results
- KD-FSNC achieves the best accuracy on all six datasets at 1, 3, and 5 shots; e.g., Cora 1-shot 87.7 vs IA-FSNC 85.5.
- Averages 1.25%, 0.82%, and 0.49% improvement over IA-FSNC at 1-, 3-, and 5-shot across datasets.
- Ablation at 3-shot: all components combined reach 92.8 on Cora vs 89.4 to 91.2 for single or partial component settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add representation distillation and pseudo-label distillation | IA-FSNC · All six datasets, 1-shot | accuracy: 1.25% | Helped |
| add representation distillation and pseudo-label distillation | IA-FSNC · All six datasets, 3-shot | accuracy: 0.82% | Helped |
| add representation distillation and pseudo-label distillation | IA-FSNC · All six datasets, 5-shot | accuracy: 0.49% | Helped |
| add local representation distillation (C1) (ablation) | KD-FSNC · Cora, 3-shot | accuracy: 89.4(0.8) -> 92.8(0.8) | Helped |
| add global representation distillation (C2) (ablation) | KD-FSNC · Cora, 3-shot | accuracy: 90.2(0.6) -> 92.8(0.8) | Helped |
| add pseudo-label distillation (C3) (ablation) | KD-FSNC · Cora, 3-shot | accuracy: 91.2(0.7) -> 92.8(0.8) | Helped |
| combine local and global representation distillation (C1+C2) (ablation) | KD-FSNC · Cora, 3-shot | accuracy: 90.6(0.7) -> 92.8(0.8) | Helped |
| combine local representation and pseudo-label distillation (C1+C3) (ablation) | KD-FSNC · Cora, 3-shot | accuracy: 90.7(0.9) -> 92.8(0.8) | Helped |
| combine global representation and pseudo-label distillation (C2+C3) (ablation) | KD-FSNC · Cora, 3-shot | accuracy: 90.5(0.6) -> 92.8(0.8) | Helped |
Surprising findings
- Local representation distillation alone performs poorly, but combined with global representation distillation it gives good results.
- Generic knowledge distillation methods (GraphAKD, SimKD) do not work well for FSNC and are sometimes below the GCN baseline.
Limitations
- Performance degrades when too many pseudo-labels or self-training cycles are used, introducing incorrect labels.
- Several hyperparameters (t, m, mu1, mu2, lambda, beta) require tuning.
Experimental setup
Datasets: Cora CiteSeer CoraFull Computers Photo Coauthor (CS)
Benchmarks: Cora CiteSeer CoraFull Computers Photo Coauthor (CS)
Metrics: classification accuracy (mean and std over 900 runs)
Baselines: GCN Meta-GNN G-META GraphAKD SimKD IA-FSNC
Builds on: IA-FSNC knowledge distillation (Hinton et al.) FitNets GCN
Closest papers by meaning
- Preference-driven Knowledge Distillation for Few-shot Node Classification (NeurIPS 2025 · similarity 0.85)
- Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data (NeurIPS 2021 · similarity 0.85)
- Cross-Level Distillation and Feature Denoising for Cross-Domain Few-Shot Classification (ICLR 2023 · similarity 0.85)
- Noisy Node Classification by Bi-level Optimization Based Multi-Teacher Distillation (AAAI 2025 · similarity 0.85)
- M2SD:Multiple Mixing Self-Distillation for Few-Shot Class-Incremental Learning (AAAI 2024 · similarity 0.85)
- Fast Graph Sharpness-Aware Minimization for Enhancing and Accelerating Few-Shot Node Classification (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.