Research map / Graph learning / GNN architectures
Pure Transformers are Powerful Graph Learners
Jinwoo Kim, Dat Tien Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, Seunghoon Hong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add node and type identifiers to node and edge tokens (ablation) | Standard Transformer · PCQM4Mv2 validation | MAE: 0.2340 -> 0.0910 | Improved |
| Use Laplacian eigenvectors instead of ORF node identifiers (ablation) | TokenGT · PCQM4Mv2 validation | MAE: 0.0962 -> 0.0910 | Improved |
| Use Performer kernel attention (ablation) | TokenGT (Lap) · PCQM4Mv2 validation | MAE: 0.0910 -> 0.0935 | Hurt |
| Add type identifiers (ablation) | Transformer self-attention · Second-order equivariant basis approximation, dense input test | L2 error: 53.93 ± 1.426 -> 40.06 ± 1.202 | Improved |
| Add node identifiers while retaining type identifiers (ablation) | Transformer self-attention · Second-order equivariant basis approximation, dense input test | L2 error: 40.06 ± 1.202 -> 0.009 ± 5e-5 | Improved |
| Use orthonormal node identifiers rather than independent first-order random embeddings (ablation) | Transformer self-attention with type identifiers · Second-order equivariant basis approximation, dense input test | L2 error: 32.49 ± 3.687 -> 0.009 ± 5e-5 | Improved |
| Use Laplacian eigenvectors instead of ORF node identifiers (ablation) | Transformer self-attention with type identifiers · Second-order equivariant basis approximation, dense input test | L2 error: 0.009 ± 5e-5 -> 0.005 ± 1e-4 | Improved |
| Use TokenGT (ORF) | PCQM4Mv2 graph regression · PCQM4Mv2 validation | MAE: 0.1083 -> 0.0962 | Improved |
| Use TokenGT (Lap) | PCQM4Mv2 graph regression · PCQM4Mv2 validation | MAE: 0.0864 -> 0.0910 | Hurt |
Closest papers by meaning
- Leveraging Contrastive Learning for Enhanced Node Representations in Tokenized Graph Transformers (NeurIPS 2024 · similarity 0.90)
- Learning Graph Quantized Tokenizers (ICLR 2025 · similarity 0.89)
- Tokenphormer: Structure-aware Multi-token Graph Transformer for Node Classification (AAAI 2025 · similarity 0.89)
- Towards Principled Graph Transformers (NeurIPS 2024 · similarity 0.89)
- Rethinking Tokenized Graph Transformers for Node Classification (NeurIPS 2025 · similarity 0.88)
- Polynormer: Polynomial-Expressive Graph Transformer in Linear Time (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
