Research map / Graph learning / Graph self-supervised learning
Scale-Free Graph-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SFGL, a graph-language model that builds a cosine-similarity KNN graph approximating a scale-free structure and uses a GNN pseudo-labeler to supervise DeBERTa finetuning for semi-supervised text-attributed graph classification.
Problem
Graph-based semi-supervised document classification suffers from artificial edge-distribution assumptions in latent graph inference and unreliable LM finetuning when labels are scarce.
Approach
Argues that a cosine-similarity KNN graph with a suitable k approximates a scale-free degree distribution, using it as a simple structural prior; a GNN trained on this graph produces pseudo-labels that supervise LM (DeBERTa) finetuning, optionally with GPT-3.5 rationales, and the process can be iterated.
Key results
- On PubMed at 0.30% labels (60 labels), SFGL+GPT reaches 0.8640 accuracy vs 0.7386 for GCN+DeBERTa+GPT and 0.4514 for DeBERTa+GPT.
- On ogbn-arxiv at 0.03% labels (50 labels), SFGL+GPT reaches 0.4570 vs 0.2137 for GCN+DeBERTa+GPT.
- Estimated power-law exponents of KNN in-degree (k=5, cosine) are 3.3 (Cora), 2.8 (PubMed), and 2.7 (ogbn-arxiv); Euclidean KNN does not fit a power law.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GPT-generated responses to SFGL text embedding | SFGL on Cora · Cora (70 labels) | accuracy: 0.6086 -> 0.6348 | Helped |
| add GPT-generated responses to SFGL text embedding | SFGL on Pubmed · Pubmed (60 labels) | accuracy: 0.7238 -> 0.8640 | Helped |
| add GPT-generated responses to SFGL text embedding | SFGL on ogbn-arxiv · ogbn-arxiv (50 labels) | accuracy: 0.3251 -> 0.4570 | Helped |
| add GPT-generated responses to SFGL text embedding | SFGL on arxiv23 · arxiv23 (50 labels) | accuracy: 0.4323 -> 0.4953 | Helped |
| use cosine rather than Manhattan distance to construct the KNN graph (ablation) | SFGL on Cora · Cora (70 labels) | accuracy: 0.5454 -> 0.6086 | Helped |
| use cosine rather than Euclidean distance to construct the KNN graph (ablation) | SFGL on Cora · Cora (70 labels) | accuracy: 0.5178 -> 0.6086 | Helped |
| use an improved GCN pseudo-labeler for an additional training iteration (ablation) | SFGL on Pubmed · Pubmed | accuracy: 0.7238 -> 0.7434; 0.7792 -> 0.8510; 0.8188 -> 0.8686; 0.8524 -> 0.8976; 0.8390 -> 0.8966 | Helped |
| use an improved GCN pseudo-labeler for an additional training iteration (ablation) | SFGL+GPT on Pubmed · Pubmed | accuracy: 0.8640 -> 0.9142; 0.9014 -> 0.9250; 0.9088 -> 0.9294; 0.9162 -> 0.9290; 0.9116 -> 0.9374 | Helped |
| replace improved GCN pseudo-labeler variant with the further-refined variant (ablation) | SFGL+GPT iterative training on Pubmed · Pubmed | accuracy: 0.9142 -> 0.9138; 0.9250 -> 0.9198; 0.9294 -> 0.9216; 0.9290 -> 0.9240; 0.9374 -> 0.9264 | Hurt |
| use KNN graph k=20 instead of the real graph (ablation) | graph-based classifier on arxiv23 · arxiv23 (50 labels) | accuracy: 0.3691 -> 0.3809 | Helped |
| increase KNN graph k from 20 to 100 (ablation) | graph-based classifier on arxiv23 · arxiv23 | accuracy: 0.3809 -> 0.3767; 0.4229 -> 0.4100; 0.4342 -> 0.4201; 0.4771 -> 0.4778; 0.4976 -> 0.5005 | Mixed |
Surprising findings
- Under very low labeling rates, a real citation graph is not significantly better and sometimes worse than a pre-constructed KNN graph on arxiv23.
- Pretrained DeBERTa, E5-Mistral-7B-Instruct and bge-en-icl embeddings perform poorly compared with a finetuned DeBERTa encoder in the text-encoder ablation.
Limitations
- Constructed KNN graphs are not exactly scale-free; the scale-free property is only approximated.
- Evaluation is limited to citation networks; social and biological networks are not studied.
- Results rely on GPT-3.5 API responses, adding cost and dependence on a closed model.
Experimental setup
Datasets: Cora PubMed ogbn-arxiv arxiv23
Benchmarks: Cora PubMed ogbn-arxiv arxiv23
Metrics: accuracy
Baselines: GCN DeBERTa GCN+DeBERTa DeBERTa+GPT GCN+DeBERTa+GPT SLAPS
Builds on: GCN DeBERTa SLAPS GPT-3.5
Closest papers by meaning
- GOFA: A Generative One-For-All Model for Joint Graph Language Modeling (ICLR 2025 · similarity 0.88)
- GLBench: A Comprehensive Benchmark for Graph with Large Language Models (NeurIPS 2024 · similarity 0.88)
- Multi-View Empowered Structural Graph Wordification for Language Models (AAAI 2025 · similarity 0.87)
- Generalization Principles for Inference over Text-Attributed Graphs with Large Language Models (ICML 2025 · similarity 0.87)
- Leveraging Large Language Models for Node Generation in Few-Shot Learning on Text-Attributed Graphs (AAAI 2025 · similarity 0.87)
- Label-free Node Classification on Graphs with Large Language Models (LLMs) (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.