Research map / Architectures and efficiency / Attention and transformers
Graph Convolutions Enrich the Self-Attention in Transformers!
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reinterprets transformer self-attention as a graph filter and proposes GFSA, which adds identity and first/K-th order matrix-polynomial terms (K-th power approximated by a first-order Taylor/finite-difference expansion) to mitigate oversmoothing across NLP, vision, graph, speech and code tasks.
Problem
Deep Transformers suffer from oversmoothing, where token representations converge across layers; standard self-attention acts as a low-pass graph filter that removes high-frequency information.
Approach
Treat the row-normalized attention matrix as a graph shift operator and use a learnable three-term matrix polynomial filter (I, A, and an approximated A^K) so the filter can be low-pass, high-pass or combined, at a cost of only tens to hundreds of extra parameters.
Key results
- GFSA improves DeiT-S top-1 accuracy on ImageNet-1k from 79.8% to 81.1% (12 layers) with under 72 extra parameters.
- RoBERTa + GFSA raises CoLA from 60.34 to 64.11 and GLUE average from 85.87 to 86.79.
- Graphormer + GFSA reduces PCQM4M validation MAE from 0.1286 to 0.1193 (about 7.2%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace self-attention with GFSA | BERT-base · GLUE (Avg) | average score: 82.51 to 83.58 | Helped |
| Replace self-attention with GFSA | ALBERT-base · GLUE (Avg) | average score: 84.01 to 85.05 | Helped |
| Replace self-attention with GFSA | RoBERTa-base · GLUE (Avg) | average score: 85.87 to 86.79 | Helped |
| Replace self-attention with GFSA | GPT2 · PTB, WikiText-2, WikiText-103 | perplexity: 18.806 to 18.764 | Helped |
| Replace self-attention with GFSA | DeiT-S, CaiT-S, and Swin-S · ImageNet-1k | top-1 accuracy: +1.63% for 12-layer DeiT | Helped |
| Replace self-attention with GFSA | Graphormer · PCQM4M validate | MAE: 0.1286 to 0.1193 | Helped |
| Replace self-attention with GFSA | Graph-ViT · Peptide-func | AP: 0.6919 to 0.6964 | Helped |
| Replace self-attention with GFSA | Transformer · LibriSpeech 100h test-clean | WER: 11.02 to 10.30 | Helped |
| Replace self-attention with GFSA | RoBERTa · Devign | accuracy: 62.88 to 64.39 | Helped |
| Use exactly calculated rather than approximated Ā^K (ablation) | BERT + GFSA · GLUE (Avg) | average score: 83.58 to 83.39 | Hurt |
| Vary polynomial order K (ablation) | BERT + GFSA · GLUE tasks | task scores | Mixed |
| Vary polynomial order K (ablation) | GPT2 + GFSA · PTB, WikiText-2, WikiText-103 | perplexity | Mixed |
Surprising findings
- Using the exact A^K instead of the Taylor approximation gave slightly lower GLUE average (83.39 vs 83.58), suggesting the approximation is not a bottleneck.
Limitations
- Adds runtime overhead from computing A^2 (quadratic in tokens) for vanilla self-attention; requires linear-attention variants for efficiency.
- Approximation of A^K by first-order Taylor expansion has only a loose theoretical error bound (2 sqrt(n) K).
- Gains are modest on several tasks (e.g., 0.31% perplexity on GPT-2, 0.12% on CodeBERT).
Experimental setup
Datasets: GLUE Penn Treebank WikiText-2 WikiText-103 ImageNet-1k LRGB Benchmarking GNNs (ZINC, MNIST, CIFAR10) OGB (ogbg-molhiv, ogbg-moltox21) PCQM4M LibriSpeech Devign
Benchmarks: GLUE WikiText-2 WikiText-103 Penn Treebank ImageNet-1k ZINC PCQM4M Peptide-func Peptide-struct MNIST CIFAR10 ogbg-molhiv
Metrics: accuracy Matthews correlation perplexity top-1 accuracy MAE AP ROCAUC word error rate
Baselines: ContraNorm AttnScale FeatScale CaiT Swin Branchformer
Builds on: ContraNorm GPR-GNN ChebNet GCN Graphormer
Closest papers by meaning
- What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding (ICML 2024 · similarity 0.90)
- Towards Principled Graph Transformers (NeurIPS 2024 · similarity 0.88)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
- Even Sparser Graph Transformers (NeurIPS 2024 · similarity 0.87)
- GrokFormer: Graph Fourier Kolmogorov-Arnold Transformers (ICML 2025 · similarity 0.87)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.