Research map / Graph learning / GNN architectures
Uncovering the Redundancy in Graph Self-supervised Learning Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that graph self-supervised GNNs (GraphMAE, GRACE, MaskGAE) are highly redundant at neuron and layer levels, and proposes SLIDE, which fine-tunes a randomly slimmed GNN plus a classifier with learned sample reweighting that decorrelates embedding dimensions.
Problem
Graph self-supervised pre-trained GNNs are usually adapted by linear probing because fine-tuning all parameters is costly; it is unclear whether all learned GNN parameters are useful.
Approach
Random removal of neurons and layers from pre-trained graph SSL models barely hurts node classification, so the slimmed GNN can be fine-tuned together with the classifier; a learnable sample-weighting scheme based on random Fourier features minimizes cross-covariance between embedding dimensions during fine-tuning.
Key results
- Randomly halving neurons in GraphMAE/GRACE keeps at least 96.2% of original performance while cutting at least 51.6% of parameters.
- SLIDE outperforms linear probing on Computers by an average of 0.75% Micro-F1 and 0.69% Macro-F1 across three pre-training frameworks.
- SLIDE achieves better performance than full fine-tuning (FT) on the evaluated datasets while fine-tuning fewer parameters; on ogbn-arxiv with GraphMAE, FT is out of memory while SLIDE runs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SLIDE instead of full fine-tuning | GraphMAE · Amazon-Computers | F1-Mi: 90.61±0.33 -> 90.85±0.34 | Helped |
| Use SLIDE instead of full fine-tuning | GraphMAE · Amazon-Computers | F1-Ma: 89.61±0.39 -> 89.88±0.36 | Helped |
| Use SLIDE instead of linear probing | GraphMAE · Ogbn-arxiv | F1-Mi: 71.90±0.09 -> 72.27±0.13 | Helped |
| Use SLIDE instead of linear probing | GraphMAE · Ogbn-arxiv | F1-Ma: 51.14±0.23 -> 53.03±0.35 | Helped |
| Use SLIDE instead of full fine-tuning | GRACE · Citeseer | F1-Mi: 70.38±0.58 -> 71.58±0.58 | Helped |
| Use SLIDE instead of full fine-tuning | GRACE · Citeseer | F1-Ma: 61.73±0.43 -> 63.63±1.03 | Helped |
| Retain 50% of neurons per layer rather than the full network (ablation) | GraphMAE · Computers | F1-Mi: 90.44 -> 89.49 | Hurt |
| Retain 25% of neurons per layer rather than the full network (ablation) | GraphMAE · Computers | F1-Mi: 90.44 -> 87.87 | Hurt |
| Retain two layers and 50% of neurons in the second layer (ablation) | GraphMAE · Computers | F1-Mi: 90.44 -> 90.14 | Hurt |
| Retain 50% of neurons per layer rather than the full network (ablation) | GRACE · Computers | F1-Mi: 87.57 -> 85.98 | Hurt |
| Retain 25% of neurons per layer rather than the full network (ablation) | GRACE · Computers | F1-Mi: 87.57 -> 83.61 | Hurt |
| Retain two layers and 50% of neurons in the second layer (ablation) | GraphMAE · IMDB-B | ACC: 75.3 -> 75.3 | No clear effect |
Surprising findings
- Randomly removing up to half or even 75% of neurons and layers after the second layer barely affects node classification accuracy of graph SSL models.
- Removing layers after the second layer has negligible impact, with 2-layer GNNs retaining 99.4% of performance on ogbn-arxiv.
Limitations
- Lacks theoretical foundations for the observed redundancy.
- Slim GNNs are obtained by random neuron removal rather than a principled pruning criterion.
- Results are limited to node classification in the main experiments; other tasks only in appendix.
Experimental setup
Datasets: Cora Citeseer Pubmed Amazon-Photo Amazon-Computers ogbn-arxiv MUTAG IMDB-B IMDB-M REDDIT-B
Benchmarks: Cora Citeseer Pubmed Amazon-Photo Amazon-Computers ogbn-arxiv
Metrics: Micro-F1 Macro-F1 accuracy AUC AP CKA parameter change
Baselines: linear probing full fine-tuning LoRA
Builds on: GraphMAE GRACE MaskGAE HSIC
Closest papers by meaning
- Decoupled Self-supervised Learning for Graphs (NeurIPS 2022 · similarity 0.89)
- Learning to Pre-train Graph Neural Networks (AAAI 2021 · similarity 0.88)
- Disentangled Graph Self-supervised Learning for Out-of-Distribution Generalization (ICML 2024 · similarity 0.87)
- Fine-Tuning Graph Neural Networks by Preserving Graph Generative Patterns (AAAI 2024 · similarity 0.86)
- Exploring Correlations of Self-Supervised Tasks for Graphs (ICML 2024 · similarity 0.86)
- HGOT: Self-supervised Heterogeneous Graph Neural Network with Optimal Transport (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.