Research map / Graph learning / Graph self-supervised learning

A Graph is Worth $K$ Words: Euclideanizing Graph using Pure Transformer

ICML 2024 · Graph learning › Graph self-supervised learning · approach: Diffusion-based graph generation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GraphsGPT, a pure-transformer encoder-decoder (Graph2Seq encoder, GraphGPT decoder) that maps molecular graphs into a fixed set of learnable Graph Words and reconstructs them, pretrained on ~100M ZINC molecules with an edge-centric GPT-style objective.

Problem

Graphs are non-Euclidean, so it is hard to encode them as Euclidean vectors that can be decoded back to the original graph, which limits autoregressive pure-transformer modeling for graph representation, generation and manipulation.

Approach

Flatten a molecular graph via DFS into a flexible token sequence, encode it into K Euclidean Graph Words with a pure transformer, and decode the graph autoregressively using an edge-centric generation scheme with block-wise causal attention and learnable position codebooks that are randomly shuffled.

Key results

• Pretrained Graph2Seq-1W achieves best results on 8 of 9 MoleculeNet tasks (e.g., BBBP 72.8, Bace 83.4, ESOL RMSD 0.860).
• Few-shot GraphGPT-1W reaches 0.995 validity and 0.995 uniqueness on MOSES, outperforming MolGPT on validity, uniqueness, novelty and diversity.
• On ZINC-250K, GraphGPT-1W (s=0.25) reports 99.67 validity and FCD 1.78, versus best baseline GDSS FCD 14.66.

Measured effects reported in the paper

Surprising findings

• Graph words from a fixed-length latent sequence allow latent-space mixup that outperforms the ICML'22 G-mixup on several MoleculeNet tasks.
• Decoding from permuted Graph Words is not fully consistent: only 96.1% average self-consistency for GraphsGPT-1W across 1024 permutations.

Limitations

• Decoder is not permutation-invariant; decoded graphs are only partially self-consistent across input orderings.
• Evaluation focuses on small molecules (ZINC, MOSES, MoleculeNet), not general graphs.
• Some comparisons (e.g., few-shot generation vs MolGPT) rely on differing settings and ZINC-250K comparisons use a reduced shot count.

Experimental setup

Datasets: ZINC MoleculeNet MOSES ZINC-C ZINC-250K GuacaMol

Benchmarks: MoleculeNet MOSES ZINC-250K ZINC-C

Metrics: ROC-AUC RMSD validity uniqueness novelty IntDiv MAD SD NSPDK FCD

Baselines: GIN InfoGraph GPT-GNN EdgePred ContextPred GraphLoG GraphMAE Mole-BERT GraphMVP MolGPT G-mixup GraphAF

Builds on: TokenGT GPT-GNN MolGPT G-mixup

Closest papers by meaning

• GraphGPT: Generative Pre-trained Graph Eulerian Transformer (ICML 2025 · similarity 0.89)
• UniGTE: Unified Graph–Text Encoding for Zero-Shot Generalization across Graph Tasks and Domains (NeurIPS 2025 · similarity 0.87)
• Learning Graph Quantized Tokenizers (ICLR 2025 · similarity 0.86)
• Graph Generative Pre-trained Transformer (ICML 2025 · similarity 0.86)
• Talk like a Graph: Encoding Graphs for Large Language Models (ICLR 2024 · similarity 0.86)
• LLaGA: Large Language and Graph Assistant (ICML 2024 · similarity 0.85)