Research map / Architectures and efficiency / Attention and transformers
Jump Self-attention: Capturing High-order Statistics in Transformers
Haoyi Zhou, Siyang Xiao, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace canonical self-attention heads with Jump Self-attention | BERT-base · GLUE dev | average score: 79.7 -> 81.7 | Improved |
| replace canonical self-attention heads with Jump Self-attention | RoBERTa-base · GLUE dev | average score: 86.4 -> 87.1 | Improved |
| add JAT to BERT-base on small dataset | BERT-base · GLUE WNLI | accuracy: 53.5 -> 57.3 | Improved |
| add JAT to RoBERTa-base | RoBERTa-base · SuperGLUE dev | average score: 75.9 -> 78.5 | Improved |
| add JAT to BERT-base | BERT-base · SQuAD v2.0 | F1: 79.3 -> 82.0 | Improved |
| add JAT to RoBERTa-base | RoBERTa-base · SQuAD v1.1 | F1: 94.6 -> 95.2 | Improved |
| use JAT in layers 1-6 versus layers 7-12 (ablation) | RoBERTa-base · GLUE CoLA | Matthews correlation: 64.6 vs 63.8 | Improved |
| use JAT heads only in layers 9-12 (ablation) | RoBERTa-base · GLUE RTE | accuracy: 78.7 -> 76.1 | Hurt |
| increase number of JAT heads from 2 to 12 in layers 1-6 (ablation) | RoBERTa-JAT · GLUE RTE | accuracy: 80.2 -> 77.8 | Hurt |
| increase number of JAT heads from 2 to 12 in layers 1-6 (ablation) | RoBERTa-JAT · GLUE CoLA | Matthews correlation: 63.7 -> 64.7 | Improved |
| use second-order instead of first-order JAT (ablation) | RoBERTa-base · GLUE RTE | accuracy: 79.9 -> 80.2 | Improved |
| use third-order instead of second-order JAT (ablation) | RoBERTa-base · GLUE RTE | accuracy: 80.2 -> 78.7 | Hurt |
Closest papers by meaning
- Graph Convolutions Enrich the Self-Attention in Transformers! (NeurIPS 2024 · similarity 0.89)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.88)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.87)
- kNN Attention Demystified: A Theoretical Exploration for Scalable Transformers (ICLR 2025 · similarity 0.87)
- CAT: Circular-Convolutional Attention for Sub-Quadratic Transformers (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
