Research map / Graph learning / GNN architectures
TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills GNN teachers into layer-wise MLP students by injecting teacher feature-transformation weights into FC layers and matching Dirichlet-energy DE ratios of each layer.
Problem
GNNs need multi-hop neighborhood data at inference, making them slow for latency-sensitive use; existing GNN-to-MLP distillation uses only soft labels and ignores layer-level knowledge of the teacher.
Approach
Treat each GNN layer's feature transformation as an MLP FC layer and inject the teacher's weights into it (with a gradient scale eta for fine-tuning), pair each graph propagation with another FC layer, and align per-layer Dirichlet energy ratios between teacher and student.
Key results
- On Citeseer (transductive), TINED reaches 74.43% accuracy versus 58.50% for MLP and 71.22% for GLNN, and is about 94x faster than the 2-layer GraphSAGE teacher (1.63ms vs 153.14ms).
- On Products under prod setting without graph dependency, TINED reaches 69.35% (prod) versus 63.47% for GLNN.
- With graph dependency, TINED-G (TINED+NOSMOG-style) reaches 75.39% on Citeseer transductive, 1.54% above NOSMOG*.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Teacher Injection (ablation) | TINED · Cora, Citeseer, Pubmed, A-computer, A-photo, Arxiv, Products transductive | accuracy: 81.32$\pm$1.60 -> 82.63$\pm$1.57; 71.88$\pm$1.27 -> 74.43$\pm$1.53; 75.93$\pm$3.40 -> 77.09$\pm$2.14; 84.59$\pm$1.78 -> | Helped |
| add Dirichlet Energy Distillation (ablation) | TINED · Cora, Citeseer, Pubmed, A-computer, A-photo, Arxiv, Products transductive | accuracy: 81.64$\pm$1.64 -> 82.63$\pm$1.57; 74.12$\pm$1.42 -> 74.43$\pm$1.53; 76.45$\pm$2.68 -> 77.09$\pm$2.14; 84.80$\pm$1.67 -> | Helped |
| use a 4-layer vanilla MLP instead of a 2-layer MLP (ablation) | vanilla MLP · Cora, Citeseer, Pubmed, A-computer, A-photo transductive | accuracy: 60.84$\pm$1.08 -> 58.20$\pm$3.16; 63.41$\pm$1.96 -> 60.81$\pm$3.48; 69.41$\pm$2.88 -> 67.61$\pm$1.51; 70.07$\pm$1.77 -> | Hurt |
| increase DE sampling ratio from 0.1 to 0.7 (ablation) | TINED · Arxiv transductive | accuracy: 64.31$\pm$0.71 -> 64.44$\pm$0.80 | No clear effect |
Surprising findings
- Feature transformations in trained GNN layers can have DE ratios above 1, i.e. they diversify rather than smooth embeddings in some layers.
- A deeper 4-layer vanilla MLP performs worse than the 2-layer MLP on several datasets, suggesting overfitting.
Limitations
- Student MLP depth is tied to the number of teacher GNN layers (2T FC layers), which the authors note as a limitation.
- Computing DE ratios on large graphs runs out of memory, requiring subgraph sampling approximation (ratio zeta).
- Gains over strong baselines are often within one standard deviation on several datasets.
Experimental setup
Datasets: Cora Citeseer Pubmed A-computer A-photo Arxiv Products Squirrel Amazon-ratings
Benchmarks: Cora Citeseer Pubmed A-computer A-photo Arxiv Products Squirrel Amazon-ratings
Metrics: accuracy inference time (ms)
Baselines: MLP GLNN GLNN* NOSMOG NOSMOG* KRD FF-G2M VQGraph GraphSAGE teacher
Builds on: GLNN NOSMOG FF-G2M KRD GraphSAGE
Closest papers by meaning
- VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs (ICLR 2024 · similarity 0.89)
- Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence (ICML 2025 · similarity 0.89)
- Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification (NeurIPS 2024 · similarity 0.89)
- Distilling Graph Geometry: Knowledge Gap from GNNs to MLPs (NeurIPS 2026 · similarity 0.89)
- Diffusing to the Top: Boost Graph Neural Networks with Minimal Hyperparameter Tuning (ICLR 2025 · similarity 0.88)
- On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.