Research map / Graph learning / GNN architectures
Graph External Attention Enhanced Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Graph External Attention (GEA), which attends to learnable shared external key-value units to capture inter-graph correlations, and GEAET, a graph Transformer that fuses GEA with message-passing GNN and self-attention layers.
Problem
Graph Transformers model only intra-graph correlations and overlook correlations between different graphs, while message-passing GNNs suffer from over-smoothing, over-squashing and limited expressivity.
Approach
Replace or complement self-attention with attention between each graph's nodes/edges and learnable external key-value units shared across all graphs (with double normalization), giving linear complexity in nodes and edges and implicitly encoding inter-graph correlations; fuse it in parallel with a GNN and a Transformer layer.
Key results
- GEAET reaches 76.634% accuracy on CIFAR10 and 98.513% on MNIST, state of the art among compared models.
- GEAET obtains F1 of 0.4585 on PascalVOC-SP and 0.3895 on COCO-SP, well above the next-best baselines (0.3966 and 0.3430 for Exphormer).
- Adding GEANet to GCN raises PATTERN accuracy from 71.892% to 85.323% and CIFAR10 from 55.710% to 61.925%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GEAET instead of GRIT | graph prediction · CIFAR10 | accuracy: 76.468 ± 0.881 -> 76.634 ± 0.427 | Helped |
| Use GEAET instead of Graph-ViT/MLPMixer | graph prediction · MNIST | accuracy: 98.460 ± 0.090 -> 98.513 ± 0.086 | Helped |
| Use GEAET instead of GRIT | graph prediction · PATTERN | accuracy: 87.196 ± 0.076 -> 86.993 ± 0.026 | Hurt |
| Use GEAET instead of Graph-ViT/MLPMixer | graph prediction · Peptides-Struct | MAE: 0.2449 ± 0.0016 -> 0.2445 ± 0.0013 | Helped |
| Use GEAET instead of Exphormer | graph prediction · PascalVOC-SP | F1 score: 0.3966 ± 0.0027 -> 0.4585 ± 0.0087 | Helped |
| Use GEAET instead of GraphGPS | graph prediction · COCO-SP | F1 score: 0.3412 ± 0.0044 -> 0.3895 ± 0.0050 | Helped |
| Add GEANet to GCN | GCN · PascalVOC-SP | F1 score: 0.1268 ± 0.0060 -> 0.2250 ± 0.0103 | Helped |
| Add GEANet to GINE | GINE · COCO-SP | F1 score: 0.1339 ± 0.0044 -> 0.2410 ± 0.0028 | Helped |
| Add GEANet to GatedGCN | GatedGCN · Peptides-Struct | MAE: 0.3420 ± 0.0013 -> 0.2547 ± 0.0009 | Helped |
| Add external node units (ablation) | GEAET without external units · MNIST | accuracy: 98.274 ± 0.011 -> 98.474 ± 0.013 | Helped |
| Add external node units (ablation) | GEAET without external units · PATTERN | accuracy: 86.882 ± 0.028 -> 86.956 ± 0.038 | Helped |
| Add external edge units to external node units (ablation) | GEAET with node units · MNIST | accuracy: 98.474 ± 0.013 -> 98.488 ± 0.082 | Helped |
Surprising findings
- Transformer performed best with a single attention head, while GEANet performed best with 8 heads.
- GEANet achieved good performance without positional encoding, whereas Transformer performed poorly without it.
Limitations
- Authors note high memory cost and time complexity remain to be reduced.
- No upper bound on expressive power is established.
- GEAET is evaluated only on graph benchmarks, not on other domains.
Experimental setup
Datasets: CIFAR10 MNIST PATTERN CLUSTER ZINC PascalVOC-SP COCO-SP Peptides-Struct Peptides-Func PCQM-Contact TreeNeighbourMatch
Benchmarks: CIFAR10 MNIST PATTERN Peptides-Struct PascalVOC-SP COCO-SP Peptides-Func PCQM-Contact ZINC CLUSTER TreeNeighbourMatch
Metrics: accuracy MAE F1 score AP MRR
Baselines: GCN GIN GINE GAT GatedGCN SAN GraphGPS GRIT Exphormer EGT
Builds on: Graph Transformer external attention (Guo et al., 2022) double normalization (Guo et al., 2021) GraphGPS GatedGCN
Closest papers by meaning
- Less is More: on the Over-Globalizing Problem in Graph Transformers (ICML 2024 · similarity 0.89)
- Towards Principled Graph Transformers (NeurIPS 2024 · similarity 0.88)
- A Scalable and Effective Alternative to Graph Transformers (AAAI 2025 · similarity 0.87)
- What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding (ICML 2024 · similarity 0.87)
- Generalizable Insights for Graph Transformers in Theory and Practice (NeurIPS 2025 · similarity 0.87)
- Unifying and Enhancing Graph Transformers via a Hierarchical Mask Framework (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.