Research map / Architectures and efficiency / Attention and transformers
Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the last Transformer attention layer with Primal-Attention | Transformer · UEA Time Series Classification (average) | accuracy: 71.9 -> 73.1 | Helped |
| Replace the last Transformer attention layer with Primal-Attention | Transformer · Long-Range Arena (average) | accuracy: 58.8 -> 60.4 | Helped |
| Replace the top-layer Decision Transformer attention with causal Primal-Attention | Decision Transformer · D4RL (average reward) | reward: 72.2±2.6 -> 77.5±6.0 | Helped |
| Replace the last DeiT-Small/16 attention layer with Primal-Attention | DeiT-Small/16 · ImageNet-100 | Top-1 accuracy: 74.2 -> 75.7 | Helped |
| Replace the last DeiT-Small/16 attention layer with Primal-Attention | DeiT-Small/16 · ImageNet-1K | Top-1 accuracy: 79.8 -> 79.8 | No clear effect |
| Replace the last Transformer attention layer with causal Primal-Attention | Transformer · WikiText-103 | perplexity: 33.0 -> 31.0 | Helped |
| Add r-scores involving the left singular vectors (ablation) | Primal. · LRA (average) | accuracy: 45.6 -> 57.5 | Helped |
| Add r-scores involving the left singular vectors (ablation) | Primal.+Trans. · LRA (average) | accuracy: 59.4 -> 60.4 | Helped |
| Vary the KSVD regularization coefficient η and projection count s (ablation) | Primal. · UEA Time Series Classification | accuracy | Mixed |
Closest papers by meaning
- Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.88)
- Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis (NeurIPS 2024 · similarity 0.88)
- Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes (ICML 2024 · similarity 0.87)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.86)
- Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level (NeurIPS 2024 · similarity 0.86)
- kNN Attention Demystified: A Theoretical Exploration for Scalable Transformers (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.