Research map / Architectures and efficiency / Attention and transformers
Random Feature Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a learned sigmoid gate to Gaussian RFA (ablation) | RFA-Gaussian · WikiText-103 small and big test | perplexity: 35.7 -> 32.7; 27.5 -> 25.0 | Helped |
| pass the last hidden state to the next mini-batch (ablation) | RFA-Gate-Gaussian · WikiText-103 test | perplexity: 32.7 -> 30.5; 25.0 -> 23.5 | Helped |
| replace Gaussian features with arc-cosine features (ablation) | RFA-Gaussian and RFA-Gate-Gaussian · WikiText-103 | perplexity: 32.7 -> 34.0; 25.0 -> 26.3 | Hurt |
| replace Gaussian features with arc-cosine features (ablation) | RFA-Gaussian · WMT14 EN-DE, EN-FR, and IWSLT14 DE-EN | BLEU: 28.0 -> 28.1; 39.2 -> 38.9; 34.5 -> 34.4 | Mixed |
| replace Gaussian random features with ELU+1 features (ablation) | RFA-Gaussian · WikiText-103 test | perplexity: 35.7 vs 40.1 | Hurt |
| replace Gaussian random features with ELU+1 features (ablation) | RFA-Gaussian · WMT14 EN-DE | BLEU: 28.0 vs 21.3 | Hurt |
| increase cross-attention feature size from 128 to 256 (ablation) | RFA-Gaussian, causal feature size 128 · WMT14 EN-DE development | BLEU: 24.9 -> 25.8 | Helped |
| increase cross-attention feature size from 256 to 512 (ablation) | RFA-Gaussian, causal feature size 128 · WMT14 EN-DE development | BLEU: 25.8 -> 26.0 | No clear effect |
| increase causal-attention feature size from 64 to 128 (ablation) | RFA-Gaussian, cross feature size 256 · WMT14 EN-DE development | BLEU: 25.3 -> 25.8 | Helped |
| increase causal-attention feature size from 128 to 256 (ablation) | RFA-Gaussian, cross feature size 256 · WMT14 EN-DE development | BLEU: 25.8 -> 25.8 | No clear effect |
| use a pool of 200 random projection matrices during training rather than one (ablation) | RFA-Gaussian · WMT14 EN-DE development | BLEU: 24.0 -> 25.8 | Helped |
| use Gaussian RFA instead of softmax attention | Transformer · WMT14 EN-DE, EN-FR, and IWSLT14 DE-EN | BLEU: 28.1 -> 28.0; 39.0 -> 39.2; 34.6 -> 34.5 | No clear effect |
Closest papers by meaning
- Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.89)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.89)
- The Closeness of In-Context Learning and Weight Shifting for Softmax Regression (NeurIPS 2024 · similarity 0.88)
- Forgetting Transformer: Softmax Attention with a Forget Gate (ICLR 2025 · similarity 0.87)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.87)
- PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.