Research map / Graph learning / Graph self-supervised learning
SSTAG: Structure-Aware Self-Supervised Learning Method for Text-Attributed Graphs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSTAG, a self-supervised pre-training framework for text-attributed graphs that co-distills an LM+GNN teacher into a structure-aware MLP student with a memory-anchor bank, enabling cross-domain transfer and cheap inference.
Problem
Graph learning models are usually trained per dataset and need many labels, so knowledge does not transfer across graphs, tasks, or domains; LLMs handle text but not structure well, and GNNs lack open-world text knowledge.
Approach
Unify node, edge, and graph tasks as PPR-sampled subgraph templates; pre-train a teacher (Sentence Transformer [CLS] tokens fused with a GCN) with masked token modeling, and distill it into an MLP student that takes [CLS] embeddings plus PPR scores, regularized by cosine alignment and a learnable memory-anchor bank.
Key results
- SSTAG reports 75.09% on Cora and 72.85% on ogbn-Arxiv node classification under linear probing, above UniGraph (74.65% and 70.89%).
- On graph classification, SSTAG reaches 82.06% ROC-AUC on BACE, versus 80.65% for the next best (Graph-LLM).
- Ablations show removing the masked loss or the GNN drops performance most (e.g., MUV 79.86 to 76.22 without L_mask; 70.57 without GNN).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SSTAG instead of UniGraph | Cross-domain node classification · Cora, Pubmed, ogbn-Arxiv, WikiCS, Products | accuracy: 74.65 ± 0.56 -> 75.09 ± 1.02; 70.84 ± 0.51 -> 72.65 ± 0.35; 70.89 ± 0.44 -> 72.85 ± 0.43; 65.47 ± 0.51 -> 68.76 ± 0.62; | Helped |
| Use SSTAG instead of the strongest baseline | Cross-domain link prediction · FB15K237, WN18RR, ML1M | ROC-AUC: 85.01 ± 0.63 -> 88.64 ± 0.49; 80.55 ± 0.2}7 -> 82.42 ± 0.66; 70.21 ± 0.51 -> 71.24 ± 0.42 | Helped |
| Use SSTAG instead of the strongest baseline | Cross-domain graph classification · HIV, BBBP, BACE, MUV | ROC-AUC: 77.27 ± 0.31 -> 79.52 ± 0.26; 73.28 ± 0.30 -> 74.38 ± 0.35; 80.65 ± 0.33 -> 82.06 ± 0.31; 77.69 ± 0.42 -> 79.86 ± 0.40 | Helped |
| Use SSTAG instead of the strongest baseline | Cross-domain graph regression · esol, LIPO, CEP | RMSE: 1.069 ± 0.006 -> 1.043 ± 0.020; 0.710 ± 0.032 -> 0.698 ± 0.003; 1.195 ± 0.012 -> 1.186 ± 0.006 | Helped |
| Add the masked modeling loss (ablation) | SSTAG · WikiCS | accuracy: 67.02 -> 68.76 | Helped |
| Add student-teacher consistency loss (ablation) | SSTAG · ogbn-Arxiv | accuracy: 71.86 -> 72.85 | Helped |
| Add memory-based consistency loss (ablation) | SSTAG · FB15K237 | ROC-AUC: 85.96 -> 88.64 | Helped |
| Use a GNN rather than removing the GNN (ablation) | SSTAG · MUV | ROC-AUC: 70.57 -> 79.86 | Helped |
| Use PPR sampling rather than neighborhood sampling (ablation) | SSTAG · WikiCS | accuracy: 68.12 -> 68.76 | Helped |
| Replace Sentence Transformer with DeBERTa-v3-base (ablation) | SSTAG · ogbn-Arxiv | accuracy: 72.85 -> 72.53 | Hurt |
| Replace Sentence Transformer with E5-large-v2 (ablation) | SSTAG · ogbn-Arxiv | accuracy: 72.85 -> 73.21 | Helped |
| Replace Sentence Transformer with LLaMA-2-7B-hf (ablation) | SSTAG · ogbn-Arxiv | accuracy: 72.85 -> 73.68 | Helped |
Surprising findings
- Replacing the GCN teacher with a much larger LLaMA-2-7B only improves ogbn-Arxiv accuracy by about 0.8 points, a modest gain for a roughly 20x larger model.
- The main results table reports the same 72.85% on ogbn-Arxiv for SSTAG and for the Graph-LLM-based variant in the cost table, so the reported numbers are not clearly distinguished.
Limitations
- Pretraining is expensive (about 22.6 h on ogbn-Arxiv cost table) and dominated by the LM teacher.
- Evaluation relies on linear probing with a single pretraining corpus (ogbn-Papers100M), so transfer to other pretraining sources is untested.
- Several reported numbers and hyperparameters are inconsistent across tables (e.g., ST listed as 110M in text but ~66M in table; Num Epochs 1).
Experimental setup
Datasets: ogbn-Papers100M Cora PubMed ogbn-Arxiv WikiCS Products FB15K237 WN18RR ML1M HIV BBBP BACE
Benchmarks: Cora PubMed ogbn-Arxiv WikiCS Products FB15K237 WN18RR ML1M HIV BBBP BACE MUV
Metrics: accuracy ROC-AUC RMSE
Baselines: GCN GIN GAT GraphCL BGRL GraphMAE GraphMAE2 Graph-LLM UniGraph
Builds on: GraphMAE GraphCL UniGraph Graph-LLM Sentence Transformers
Closest papers by meaning
- Generalization Principles for Inference over Text-Attributed Graphs with Large Language Models (ICML 2025 · similarity 0.91)
- UniGTE: Unified Graph–Text Encoding for Zero-Shot Generalization across Graph Tasks and Domains (NeurIPS 2025 · similarity 0.90)
- Unifying Text Semantics and Graph Structures for Temporal Text-attributed Graphs with Large Language Models (NeurIPS 2025 · similarity 0.89)
- Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning (ICLR 2024 · similarity 0.89)
- Large Language Model Meets Graph Neural Network in Knowledge Distillation (AAAI 2025 · similarity 0.88)
- Leveraging Large Language Models for Node Generation in Few-Shot Learning on Text-Attributed Graphs (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.