Research map / Architectures and efficiency / Attention and transformers
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use normalized PRF attention with RPE | Transformer pre-training · GLUE dev AVG | score: 84.4 -> 85.2 | Helped |
| Use normalized PRF attention with RPE | Transformer language modeling · WikiText-103 validation | perplexity: 31.3 -> 30.6 | Helped |
| Replace standard attention with NPRF attention and RPE in both encoder and decoder | Transformer encoder-decoder · IWSLT14 German-English, French-English | BLEU: 34.7 -> 34.3; 29.2 -> 29.4; 40.0 -> 39.9; 40.2 -> 40.4 | Mixed |
| Use normalized PRF attention with RPE | DeiT-base · ImageNet-1K | Top-1 accuracy: 81.2 -> 80.9 | Hurt |
| Use normalized PRF attention with RPE instead of PRF attention | Transformer image generation model · ImageNet32 | BPD: 4.04 -> 3.68 | Helped |
| Add RPE to normalized PRF attention (ablation) | DeiT-base · ImageNet-1K | Top-1 accuracy: 77.7 -> 80.9 | Helped |
| Add RPE to normalized PRF attention (ablation) | DeiT-base · ImageNet-1K | Top-5 accuracy: 93.3 -> 94.8 | Helped |
| Use PRF attention in both encoder and decoder without RPE or query/key normalization (ablation) | Transformer encoder-decoder · IWSLT14 De-En | BLEU: 34.7 -> 32.1 | Hurt |
| Apply PRF attention only in the decoder (ablation) | Transformer encoder-decoder · IWSLT14 De-En | BLEU: 34.7 -> 34.6 | No clear effect |
| Normalize queries and keys before kernel feature mapping (ablation) | Transformer attention converted to kernelized attention · IWSLT14 De-En validation | BLEU | Helped |
| Add RPE to attention before conversion to kernelized attention (ablation) | Transformer attention conversion · IWSLT14 De-En validation | BLEU | Helped |
| Vary the feature-map dimension (ablation) | NPRF-Transformer with RPE · IWSLT14 De-En validation | BLEU | No clear effect |
Closest papers by meaning
- RelFlexformer: Efficient Attention Transformers for Integrable Relative Positional Encodings (NeurIPS 2026 · similarity 0.90)
- MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention (NeurIPS 2025 · similarity 0.89)
- CAT: Circular-Convolutional Attention for Sub-Quadratic Transformers (NeurIPS 2025 · similarity 0.88)
- Positional Attention: Expressivity and Learnability of Algorithmic Computation (ICML 2025 · similarity 0.87)
- PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels (ICML 2024 · similarity 0.87)
- kNN Attention Demystified: A Theoretical Exploration for Scalable Transformers (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.