Research map / Graph learning / Graph self-supervised learning
Graph Generative Pre-trained Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes G2PT, an autoregressive Transformer that generates graphs by next-token prediction over a node-and-edge token sequence, extended to goal-oriented generation (RFT, PPO) and molecular property prediction via fine-tuning.
Problem
Graph generation is usually done with adjacency matrices, which need quadratic computation and, for diffusion models, permutation-invariant networks and many denoising steps. The paper asks whether a compact edge-sequence representation with a generic autoregressive Transformer can match or beat the
Approach
Represent a graph as a token sequence listing all nodes (type, index) followed by edges (source, destination, type) in a degree-based edge order, and train a GPT-style decoder on it with next-token prediction. The sequence model is then adapted with rejection sampling fine-tuning and PPO for goal-oriented generation and with a classification head for property prediction.
Key results
- On MOSES, G2PT_large reaches 97.2% validity and G2PT_base reaches 0.97 FCD, the best among compared models on those metrics.
- On the generic datasets, G2PT_base achieves 11 of 24 best scores and ranks top-two on 17 of 24 metrics.
- On eight MoleculeNet classification tasks, G2PT_base averages 73.3 ROC-AUC, matching GraphMAE's 73.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the G2PT edge-sequence representation instead of an adjacency-matrix sequence | Transformer graph generation on Planar · Planar | Deg. MMD: 8.6e-3 -> 4.7e-3 | Helped |
| pre-train G2PT on GuacaMol before MoleculeNet fine-tuning (ablation) | G2PT-base · MoleculeNet average | ROC-AUC: 64.9 -> 73.3 | Helped |
| increase model size from small to base | G2PT · MOSES | Validity: 95.1 -> 96.4 | Helped |
| increase model size from base to large | G2PT · MOSES | Validity: 96.4 -> 97.2 | Helped |
| increase model size from base to large | G2PT · MOSES | Novelty: 86.0 -> 79.4 | Hurt |
| use the full Transformer rather than only the linear layer for prediction fine-tuning (ablation) | G2PT property prediction · MoleculeNet | ROC-AUC | Helped |
| use multiple sequences per graph for data augmentation rather than one sequence (ablation) | G2PT pre-training · molecular datasets | validity | Helped |
| use PPO fine-tuning instead of rejection-sampling fine-tuning | G2PT goal-oriented generation · GSK3β | target reward distribution | Hurt |
Surprising findings
- Unconstrained logits (no masking of illegal tokens) still yield superior performance, despite illegal tokens being possible at each step.
- On GSK3beta, PPO fails to sample high-reward molecules while RFT with bootstrapping succeeds.
Limitations
- G2PT is order-sensitive, so results depend on the chosen node and edge ordering.
- Does not maintain permutation invariance.
Experimental setup
Datasets: QM9 MOSES GuacaMol Planar Tree Lobster SBM MoleculeNet
Benchmarks: Planar Tree Lobster SBM QM9 MOSES GuacaMol MoleculeNet
Metrics: MMD degree/clustering/orbit/spectral/wavelet V.U.N. validity FCD ROC-AUC
Baselines: GRAN BiGG DiGress BwR HSpectre GEEL DeFoG DisCo Cometh GraphMAE
Builds on: GraphRNN DeepGMG PPO GraphMAE
Closest papers by meaning
- GraphGPT: Generative Pre-trained Graph Eulerian Transformer (ICML 2025 · similarity 0.93)
- Generative Graph Pattern Machine (NeurIPS 2025 · similarity 0.90)
- Graph Generation with $K^2$-trees (ICLR 2024 · similarity 0.89)
- A Simple and Scalable Representation for Graph Generation (ICLR 2024 · similarity 0.88)
- Homomorphism Counts as Structural Encodings for Graph Learning (ICLR 2025 · similarity 0.88)
- Leveraging Contrastive Learning for Enhanced Node Representations in Tokenized Graph Transformers (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.