Research map / Architectures and efficiency / Attention and transformers
Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models
Difan Deng, Andreas Bentzen Winje, Lukas Fehring, Marius Lindauer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add softmax attention in inner-chunk computation alongside GDN (ablation) | NAtS-L Hybrid · Retrieval tasks avg | avg accuracy: 26.98 -> 33.49 | Improved |
| add GDN in inner-chunk computation alongside softmax attention (ablation) | NAtS-L Hybrid · Retrieval tasks avg | avg accuracy: 30.87 -> 33.49 | Improved |
| add decay of hidden states for softmax chunks (ablation) | NAtS-L Hybrid · Retrieval tasks avg | avg accuracy: 32.24 -> 33.49 | Improved |
| add separate normalization of each attention output before summing (ablation) | NAtS-L Hybrid · Retrieval tasks avg | avg accuracy: 29.59 -> 33.49 | Improved |
| add output weights for combining attention operations (ablation) | NAtS-L Hybrid · Retrieval tasks avg | avg accuracy: 32.34 -> 33.49 | Improved |
| use softmax instead of sigmoid for output weights (ablation) | NAtS-L Hybrid · Retrieval tasks avg | avg accuracy: 30.02 -> 33.49 | Improved |
| use softmax instead of linear output weights (ablation) | NAtS-L Hybrid · Retrieval tasks avg | avg accuracy: 30.57 -> 33.49 | Improved |
| compute output weights from q instead of input X (ablation) | NAtS-L Hybrid · Retrieval tasks avg | avg accuracy: 27.64 -> 33.49 | Improved |
| use chunk size 64 instead of 128 (ablation) | NAtS-L Hybrid · Retrieval tasks avg | avg accuracy: 27.85 -> 33.49 | Improved |
| insert NAtS-L layers into GDN layers (hybrid) versus NAtS-L only | NAtS-L · Retrieval tasks avg (800M) | avg accuracy: 39.75 -> 45.07 | Improved |
| NAtS-L Hybrid versus layer-wise GDN Hybrid | GDN Hybrid · Retrieval tasks avg (800M) | avg accuracy: 37.30 -> 45.07 | Improved |
| NAtS-L Hybrid versus GDN Hybrid at longer context | GDN Hybrid · RULER 16k | mean score: 0.00 -> 0.21 | Improved |
Closest papers by meaning
- Neural Attention Search (NeurIPS 2025 · similarity 0.91)
- MVA: Linear Attention with High-order Query-Keys Integration and Multi-level Vocabulary Decomposition (ICML 2025 · similarity 0.88)
- STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs (NeurIPS 2026 · similarity 0.88)
- Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · similarity 0.88)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.88)
- The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
