Research map / Graph learning / GNN architectures
NAAM: Node-Aware Attention Mechanism for Distilling GNNs-to-MLP (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NAAM, a node-aware attention mechanism that assigns per-node attention weights over multiple teacher GNNs when distilling them into an MLP student.
Problem
Distilling multiple GNN teachers into an MLP student, where existing methods (e.g., MTAAM) assign one attention weight per teacher GNN, assuming equal importance across all nodes.
Approach
Compute attention per node rather than per teacher: each node gets softmax weights over K teachers, scored by negative KL divergence between learnable linear transforms of student and teacher outputs, optimized each epoch with a validation CE loss plus a smoothing loss on attention changes.
Key results
- NAAM improves over MTAAM by up to 4.40% in the inductive setting on Cora (75.0 vs 71.2 transductive/product comparisons reported as relative gains per setting).
- NAAM reaches 82.9% on A-computer (prod) versus 81.9% for MTAAM and 82.0% for GLNN.
- Figure 2 reports NAAM is more accurate than other KD methods and faster than GNN teachers on average across datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use node-aware teacher attention instead of one attention weight per teacher | MTAAM-style GNN-to-MLP distillation · Cora prod | accuracy: 73.5 -> 75.0 | Helped |
| Use node-aware teacher attention instead of one attention weight per teacher | MTAAM-style GNN-to-MLP distillation · Cora ind | accuracy: 68.2 ± 1.7 -> 71.2 ± 1.8 | Helped |
| Use node-aware teacher attention instead of one attention weight per teacher | MTAAM-style GNN-to-MLP distillation · Cora tran | accuracy: 78.7 ± 2.2 -> 78.8 ± 2.2 | Helped |
| Use node-aware teacher attention instead of one attention weight per teacher | MTAAM-style GNN-to-MLP distillation · Citeseer prod | accuracy: 70.0 -> 71.4 | Helped |
| Use node-aware teacher attention instead of one attention weight per teacher | MTAAM-style GNN-to-MLP distillation · Citeseer ind | accuracy: 69.3 ± 1.9 -> 70.8 ± 2.3 | Helped |
| Use node-aware teacher attention instead of one attention weight per teacher | MTAAM-style GNN-to-MLP distillation · Citeseer tran | accuracy: 70.6 ± 2.0 -> 71.9 ± 2.3 | Helped |
| Use node-aware teacher attention instead of one attention weight per teacher | MTAAM-style GNN-to-MLP distillation · Pubmed prod | accuracy: 75.2 -> 76.4 | Helped |
| Use node-aware teacher attention instead of one attention weight per teacher | MTAAM-style GNN-to-MLP distillation · Pubmed ind | accuracy: 75.1 ± 2.1 -> 76.3 ± 2.2 | Helped |
| Use node-aware teacher attention instead of one attention weight per teacher | MTAAM-style GNN-to-MLP distillation · Pubmed tran | accuracy: 75.4 ± 2.4 -> 76.5 ± 2.4 | Helped |
| Use node-aware teacher attention instead of one attention weight per teacher | MTAAM-style GNN-to-MLP distillation · A-computer prod | accuracy: 81.9 -> 82.9 | Helped |
| Use node-aware teacher attention instead of one attention weight per teacher | MTAAM-style GNN-to-MLP distillation · A-computer ind | accuracy: 80.6 ± 1.6 -> 81.5 ± 1.4 | Helped |
| Use node-aware teacher attention instead of one attention weight per teacher | MTAAM-style GNN-to-MLP distillation · A-computer tran | accuracy: 83.3 ± 2.0 -> 84.3 ± 1.6 | Helped |
Limitations
- Gains on some settings are small (e.g., A-photo transductive +0.22%, inductive 0.00%).
- Experiments limited to five citation/co-purchase node classification datasets with a fixed set of five GNN teachers.
Experimental setup
Datasets: Cora Citeseer Pubmed A-computer A-photo
Benchmarks: Cora Citeseer Pubmed A-computer A-photo
Metrics: test accuracy inference time
Baselines: GLNN MTAAM GCN GAT APPNP SGC GCNII
Builds on: GLNN MTAAM
Closest papers by meaning
- TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillation (ICML 2025 · similarity 0.86)
- Large Language Model Meets Graph Neural Network in Knowledge Distillation (AAAI 2025 · similarity 0.84)
- VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs (ICLR 2024 · similarity 0.84)
- Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Data (ICML 2025 · similarity 0.84)
- Exploring Rationale Learning for Continual Graph Learning (AAAI 2025 · similarity 0.84)
- Preference-driven Knowledge Distillation for Few-shot Node Classification (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.