Research map / Architectures and efficiency / RNNs and normalization
LightHGNN: Distilling Hypergraph Neural Networks into MLPs for 100x Faster Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills hypergraph neural network teachers (HGNN) into MLP students via soft labels plus reliable-hyperedge high-order soft-label loss, enabling hypergraph-free inference about 100x faster.
Problem
HGNN inference is slow and memory-heavy because it needs high-order hypergraph message passing, which hinders industrial deployment; MLPs are fast but lose accuracy on hypergraph tasks.
Approach
Distill HGNN teachers into MLP students using soft labels (LightHGNN), and additionally inject high-order knowledge by sampling reliable hyperedges (scored by entropy-variance under feature noise) and matching hyperedge-level soft targets (LightHGNN+).
Key results
- LightHGNN+ outperforms vanilla MLP by about 16.3 points on average across eight hypergraph datasets under transductive setting, with only about 0.3-0.4 average drop versus HGNN.
- On synthetic hypergraphs with 55,000 vertices, LightHGNNs run inference in 1.37 ms versus 137.70 ms for HGNN (about 100x faster).
- Under the inductive setting on DBLP-Conf, LightHGNN drops about 11 points relative to HGNN, showing a limitation for unseen topology.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill HGNN soft labels into an MLP (LightHGNN) | MLP · eight hypergraph datasets, transductive | accuracy: 16.33 | Helped |
| distill HGNN knowledge into an MLP instead of using HGNN | HGNN · eight hypergraph datasets, transductive | accuracy: −0.29 | Mixed |
| add topology-aware high-order soft-target distillation (LightHGNN+) | LightHGNN · hypergraph datasets, transductive | accuracy | Mixed |
| add topology-aware high-order soft-target distillation (ablation) | LightHGNN · IMDB-AW, 2–10 layers | accuracy | Helped |
| increase the feature-noise ratio at evaluation (ablation) | LightHGNN and LightHGNN+ · DBLP-Paper | accuracy | Hurt |
| increase training samples per class from 5 to 500 (ablation) | MLP, HGNN, LightHGNN, and LightHGNN+ · DBLP-Paper, transductive | accuracy | Mixed |
| decrease λ, reducing the weight of soft-label supervision (ablation) | LightHGNN and LightHGNN+ · DBLP-Paper | accuracy | Hurt |
| vary distillation temperature τ away from 0.5 (ablation) | LightHGNN+ · DBLP-Paper | accuracy | Hurt |
| increase the number of layers (ablation) | MLP, HGNN, LightHGNN, and LightHGNN+ · IMDB-AW | accuracy | Mixed |
| use different teacher HGNN architectures (ablation) | LightHGNN and LightHGNN+ · DBLP-Paper, production setting | accuracy | Mixed |
| increase the inductive testing ratio (ablation) | MLP, HGNN, LightHGNN, and LightHGNN+ · DBLP-Paper, production setting | accuracy | Hurt |
Surprising findings
- Under the inductive setting, LightHGNNs fall notably behind HGNN on some datasets (e.g., DBLP-Conf, about 11% decline), since they cannot use unseen topology at inference.
- On DBLP-Conf, LightHGNN and LightHGNN+ underperform the teacher HGNN by roughly 3.8 to 4 points in the transductive setting.
Limitations
- Performance drops relative to HGNN under the inductive setting because student MLPs cannot exploit unseen topology.
- Gap to teacher HGNN remains on some datasets (e.g., DBLP-Conf, DBLP-Term), and LightHGNN+ does not uniformly beat LightHGNN.
- Over-smoothing analysis and the reliability score rely on heuristic design choices (noise distribution, temperature tau=0.5) with limited theoretical grounding.
Experimental setup
Datasets: News20 CA-Cora CC-Cora CC-Citeseer DBLP-Paper DBLP-Term DBLP-Conf IMDB-AW Cora Pubmed Citeseer Recipe-100k
Benchmarks: News20 CA-Cora CC-Cora CC-Citeseer DBLP-Paper DBLP-Term DBLP-Conf IMDB-AW Cora Pubmed Citeseer Recipe-100k
Metrics: accuracy inference time (ms) memory (MB) topology-aware score
Baselines: MLP GCN GAT HGNN HGNN+ HNHN GLNN KRD NOSMOG
Builds on: GLNN KRD Hinton knowledge distillation HGNN HGNN+
Closest papers by meaning
- Training-Free Message Passing for Learning on Hypergraphs (ICLR 2025 · similarity 0.89)
- When Hypergraph Meets Heterophily: New Benchmark Datasets and Baseline (AAAI 2025 · similarity 0.89)
- Deep Hypergraph Neural Networks with Tight Framelets (AAAI 2025 · similarity 0.88)
- Hypergraph Neural Architecture Search (AAAI 2024 · similarity 0.88)
- SPHINX: Structural Prediction using Hypergraph Inference Network (ICML 2025 · similarity 0.88)
- K-hop Hypergraph Neural Network: A Comprehensive Aggregation Approach (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.