Research map / Graph learning / Graph self-supervised learning
Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing graph learning benchmarks must shift toward real-world applications, meaningful graph construction, and rigorous evaluation, supported by small empirical re-evaluations of baselines and a multi-task pretraining experiment.
Problem
Graph learning benchmarks focus on narrow, saturated, or poorly constructed datasets (e.g., 2D molecules, ZINC, citation graphs) with inconsistent evaluation protocols, which hinders real-world impact and the development of graph foundation models.
Approach
Argues that graph learning needs transformative real-world benchmarks (combinatorial optimization, relational databases, chip design), meaningful graph construction with structure-free baselines, and rigorous evaluation; supports this with re-evaluations showing baseline GNNs are under-tuned and a multi-task encoder-processor-decoder pretraining study.
Key results
- On PCQM4Mv2, re-tuned 20-layer GINE (~20M params) reduces validation MAE from ~0.12 to 0.0913 (0.0898 with RWSE), competitive with graph transformers (0.0832-0.0890).
- Heterophilous baselines GCN/SAGE/GAT re-evaluated without the added MLP modifications drop substantially, with average gains of +43.56%, +14.63%, +19.49% attributed to the architectural changes on roman-empire, amazon-ratings, minesweeper.
- On 5 of 9 OGB experiments, a non-informative fixed Cayley expander graph matched or outperformed the original graph; on molbbbp GraphConv without any graph (DeepSets) reached 64.90 vs 64.11 with original graph in the table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the original graph with a fixed Cayley graph (ablation) | GraphConv, GIN, and GAT · molhiv, molbbbp, and molbace | ROC-AUC: GraphConv: molhiv 68.24 ± 1.77 -> 67.91 ± 0.75; molbbbp 64.11 ± 4.50 -> 61.60 ± 4.48; molbace 63.18 ± 4.56 -> 56.94 ± 7. | Mixed |
| Drop graph structure and use DeepSet (ablation) | GraphConv on molbbbp · molbbbp | ROC-AUC: 64.11 ± 4.50 -> 64.90 ± 0.72 | Helped |
| Retune GINE using a 20-layer, approximately 20M-parameter configuration | GINE · PCQM4Mv2 validation | MAE: 0.1195 -> 0.0913 | Helped |
| Add RWSE node features (ablation) | Retuned GINE · PCQM4Mv2 validation | MAE: 0.0913 ± 0.0002 -> 0.0898 ± 0.0001 | Helped |
| Add the reported linear encoder, two-layer MLP after each aggregation, linear decoder, and GeLU activations (ablation) | GCN, SAGE, and GAT · roman-empire, amazon-ratings, and minesweeper | accuracy: Avg. % Gain: +43.56%; +14.63%; +19.49% | Helped |
| Multi-task train instead of single-task train | MPNN and GT · COCO-SP | F1: MPNN 0.0817 -> 0.0413; GT 0.2947 -> 0.1137 | Hurt |
| Multi-task train instead of single-task train | MPNN and GT · MalNetTiny | accuracy: MPNN 81.80 -> 83.20; GT 81.90 -> 88.90 | Helped |
Surprising findings
- A fixed, structure-uninformative Cayley expander graph matched or beat the original molecular graph on several OGB tasks.
- Simply re-tuning a plain GINE model cut reported PCQM4Mv2 error by over 20%, closing much of the claimed graph transformer gap.
Limitations
- Empirical experiments are small-scale and use a single seed for the multi-task pretraining study.
- Downstream transfer results are mixed (negative transfer for MPNN on Peptides-struct).
- Position paper; many recommendations are not empirically validated.
Experimental setup
Datasets: PCQM4Mv2 COCO-SP MalNetTiny PascalVOC-SP Peptides-struct Stargazers ogbg-molhiv ogbg-molbbbp ogbg-molbace roman-empire amazon-ratings minesweeper
Benchmarks: OGB molhiv OGB molbbbp OGB molbace PCQM4Mv2 roman-empire amazon-ratings minesweeper COCO-SP MalNetTiny PascalVOC-SP Peptides-struct Stargazers
Metrics: ROC-AUC MAE accuracy F1 test performance vs fine-tuning steps
Baselines: GCN GINE GAT GraphConv GPS Graphormer DeepSets TokenGT GRPE ET
Builds on: message-passing neural networks graph transformers Open Graph Benchmark Long Range Graph Benchmark GraphWorld
Closest papers by meaning
- No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets (ICML 2025 · similarity 0.89)
- On the Scalability of GNNs for Molecular Graphs (NeurIPS 2024 · similarity 0.89)
- Empowering Active Learning for 3D Molecular Graphs with Geometric Graph Isomorphism (NeurIPS 2024 · similarity 0.87)
- B-XAIC Dataset: Benchmarking Explainable AI for Graph Neural Networks Using Chemical Data (NeurIPS 2026 · similarity 0.86)
- Core Knowledge Learning Framework for Graph (AAAI 2025 · similarity 0.86)
- Knowledge Enhanced Representation Learning for Drug Discovery (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.