Research map / Graph learning / Graph self-supervised learning
GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GraphGPT, a generative pre-trained transformer for graphs that serializes (sub)graphs into Eulerian-path token sequences and pre-trains with next-token or scheduled masked-token prediction, reporting SOTA-level results on several OGB tasks and scaling to 2B parameters.
Problem
Graph transformers rely on handcrafted structural features or GNN modules, struggle with edge- and node-level tasks, and lack effective generative pre-training frameworks that scale to large graphs.
Approach
Losslessly and reversibly convert graphs or sampled subgraphs into token sequences via (semi-)Eulerian paths (with Eulerization, cyclic node re-indexing, and multi-token node identity encoding), then apply a standard Llama-style transformer encoder or decoder pre-trained with NTP or SMTP and fine-tuned with a unified task formatting.
Key results
- On PCQM4Mv2, GraphGPT-B48 reaches a test MAE of 0.0804, improving on the previous best 0.0821 (GPTrans-L).
- On ogbl-ppa, GraphGPT-XXL reaches HR@100 of 76.55% versus 73.74% for RefinedGAE, and the model scales to 2B parameters.
- On ogbn-proteins, GraphGPT-B reaches ROC-AUC 85.33% using subgraphs of about 40 nodes, while GNNs like AGDN need subgraphs with over 22,000 nodes; on ogbn-arxiv it gets 72.10%, below the best GNNs (about 74.16%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GraphGPT-B48 instead of GPTrans-L | PCQM4Mv2 · PCQM4Mv2 test | MAE: 0.0821 -> 0.0804 | Helped |
| Use GraphGPT-B24 instead of GPTrans-L† | ogbg-molpcba · ogbg-molpcba test | Average Precision (%): 32.43_{\pm 0.22} -> 31.81_{\pm 0.1} | Hurt |
| Use GraphGPT-B with combined Triangles, Reddit-threads, and Internal pre-training | Triangles · Triangles T-large | Accuracy (%): 13.99_{\pm 1.78} -> 58.96_{\pm 1.90} | Helped |
| Use GraphGPT-XXL instead of RefinedGAE | ogbl-ppa · ogbl-ppa | HR@100 (%): 73.74_{\pm 0.92} -> 76.55_{\pm 0.67} | Helped |
| Use GraphGPT-B instead of MPLP | ogbl-citation2 · ogbl-citation2 | MRR (%): 90.72_{\pm 0.12} -> 93.05_{\pm 0.20} | Helped |
| Use GraphGPT-B instead of AGDN | ogbn-proteins · ogbn-proteins | ROC-AUC (%): 88.65_{\pm 0.13} -> 85.33_{\pm 0.10} | Hurt |
| Use GraphGPT-B instead of DRGAT | ogbn-arxiv · ogbn-arxiv | Accuracy (%): 74.16_{\pm 0.07} -> 72.10_{\pm 0.30} | Hurt |
| Add SMTP pre-training instead of no pre-training with decoder (ablation) | GraphGPT on PCQM4Mv2 · PCQM4Mv2 valid | MAE: 0.0978 -> 0.0807 | Helped |
| Use SMTP instead of NTP pre-training (ablation) | GraphGPT on PCQM4Mv2 · PCQM4Mv2 valid | MAE: 0.0875 -> 0.0807 | Helped |
| Use SMTP instead of NTP pre-training (ablation) | GraphGPT on ogbg-molpcba · ogbg-molpcba test | Average Precision: 23.85 -> 27.56 | Helped |
| Add PCQM4Mv2 regression fine-tuning after multi-dataset SMTP pre-training (ablation) | GraphGPT on ogbg-molpcba · ogbg-molpcba test | Average Precision: 27.20 -> 28.07 | Helped |
| Use SMTP instead of NTP pre-training (ablation) | GraphGPT on ogbl-ppa · ogbl-ppa test | HR@100: 55.56 -> 55.68 | No clear effect |
Surprising findings
- Node re-indexing increases pre-training loss yet improves downstream performance on ogbg-molpcba.
- On ogbn-arxiv, GraphGPT lags behind traditional GNNs and some graph transformers, which the authors attribute to dataset size.
Limitations
- Pre-training is dataset-specific, limiting transfer across domains with different semantics.
- Performance on small to medium datasets such as ogbn-arxiv lags behind traditional GNNs.
- Pre-training large graphs or PCQM4Mv2 with 50M+ parameters is computationally expensive.
Experimental setup
Datasets: PCQM4Mv2 ogbg-molpcba Triangles ogbl-ppa ogbl-citation2 ogbn-proteins ogbn-arxiv Reddit-threads Erdos-Renyi random graphs
Benchmarks: PCQM4Mv2 ogbg-molpcba Triangles ogbl-ppa ogbl-citation2 ogbn-proteins ogbn-arxiv
Metrics: MAE average precision accuracy HR@100 MRR ROC-AUC
Baselines: GCN GIN GCN-VN GIN-VN TokenGT Graphformer GPS-Deep GPTrans-L AGDN SEAL MPLP RefinedGAE
Builds on: Llama BERT ShaDowKHop METIS NetworkX
Closest papers by meaning
- Graph Generative Pre-trained Transformer (ICML 2025 · similarity 0.93)
- Triplet Interaction Improves Graph Transformers: Accurate Molecular Graph Learning with Triplet Graph Transformers (ICML 2024 · similarity 0.89)
- A Graph is Worth $K$ Words: Euclideanizing Graph using Pure Transformer (ICML 2024 · similarity 0.89)
- Generative Graph Pattern Machine (NeurIPS 2025 · similarity 0.88)
- VCR-Graphormer: A Mini-batch Graph Transformer via Virtual Connections (ICLR 2024 · similarity 0.87)
- Graph Positional and Structural Encoder (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.