Research map / Architectures and efficiency / Attention and transformers
Gated Slot Attention for Efficient Linear-Time Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Gated Slot Attention (GSA), a gated version of ABC reformulated as a two-pass GLA linked by softmax, trained with hardware-efficient chunkwise kernels and evaluated in from-scratch LM and Mistral-7B T2R finetuning settings.
Problem
Linear attention and gated linear RNNs lag behind softmax Transformers on recall-intensive in-context tasks, and training them from scratch is expensive; T2R finetuning of softmax Transformers into linear models suffers from a softmax-to-kernel discrepancy.
Approach
Reformulate ABC as a two-pass linear attention with a softmax in between, add data-dependent forget gates per slot (GSA), so it can reuse GLA's chunkwise training kernel and keep softmax for T2R finetuning, with a small recurrent state.
Key results
- At 1.3B/100B tokens, GSA averages 51.7 on zero-shot commonsense tasks versus 50.4 for GLA and 50.6 for RetNet, with a 128×Ld state versus 256×Ld for GLA.
- Mistral-7B finetuned to GSA with 100B tokens averages 56.9 across the reported 7B benchmarks, above SUPRA (54.2) and RWKV6 (54.1) and Mamba (55.7).
- On long-context tasks with 20B finetuning tokens, GSA scores 19.2 on NarrativeQA versus 17.2 for GLA and 0.0 for RetNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add input-dependent decay gating to ABC (ablation) | ABC on Slimpajama · Slimpajama, 340M, 10B tokens | PPL: 16.94 -> 13.51 | Helped |
| Use input-dependent rather than data-independent decay (ablation) | GSA on Slimpajama · Slimpajama, 340M, 10B tokens | PPL: 15.83 -> 13.51 | Helped |
| Retain softmax rather than removing it (ablation) | GSA on Slimpajama · Slimpajama, 340M, 10B tokens | PPL: 14.03 -> 13.51 | Helped |
| Replace softmax with Swish (ablation) | GSA on Slimpajama · Slimpajama, 340M, 10B tokens | PPL: 13.51 -> 13.71 | Hurt |
| Replace softmax with ReLU (ablation) | GSA on Slimpajama · Slimpajama, 340M, 10B tokens | PPL: 13.51 -> 13.69 | Hurt |
| Replace softmax with ReLU^2 (ablation) | GSA on Slimpajama · Slimpajama, 340M, 10B tokens | PPL: 13.51 -> 13.95 | Hurt |
| Reduce slot count from 64 to 32 (ablation) | GSA on Slimpajama · Slimpajama, 340M, 10B tokens | PPL: 13.51 -> 13.74 | Hurt |
| Increase slot count from 64 to 128 (ablation) | GSA on Slimpajama · Slimpajama, 340M, 10B tokens | PPL: 13.51 -> 13.46 | Helped |
| Finetune Mistral 7B into GSA rather than GLA | Mistral 7B · Commonsense, knowledge, and aggregated benchmarks; +20B tokens | Avg.: 52.5 -> 53.9 | Helped |
| Finetune Mistral 7B into GSA rather than SUPRA | Mistral 7B · Commonsense, knowledge, and aggregated benchmarks; +100B tokens | Avg.: 54.2 -> 56.9 | Helped |
| Finetune Mistral 7B into GSA rather than GLA | Mistral 7B · Long-context Qasper, NarrativeQA, QuALITY, and QMSum | Score | Helped |
| Use GSA instead of HGRN2 | 1.3B model · Commonsense reasoning, 100B tokens | Avg. accuracy: 51.9 -> 51.7 | Hurt |
Surprising findings
- GSA shows negative results on Phonebook lookup and Needle-in-Haystack compared with Transformers, indicating limits on precise copy-style retrieval.
- Removing softmax in the first pass but keeping the other design choices gave a slightly better perplexity (13.71 with Swish) than the softmax variant (13.51 with softmax was best overall, so softmax helps), while the ablation shows the data-dependent gate matters most.
Limitations
- Pretrained models are small-scale (1.3B/2.7B, 100B tokens), so long-context results are limited and not reported for those models.
- Weaker than Transformers on copy-oriented tasks such as Phonebook lookup and Needle-in-Haystack.
- Did not explore a two-pass DeltaNet variant or compare against newer RNNs like xLSTM, Mamba2, and TTT.
Experimental setup
Datasets: SlimPajama MQAR FDA SWDE SQuAD NQ TriviaQA Drop Qasper NarrativeQA QuALITY QMSum
Benchmarks: ARC-easy ARC-challenge HellaSwag LAMBADA PIQA WikiText WinoGrande MQAR FDA SWDE SQuAD NQ
Metrics: perplexity accuracy F1 training throughput inference latency
Baselines: Transformer++ (Xfmr++) Mamba RetNet GLA HGRN2 RWKV6 SUPRA Mistral-7B
Builds on: ABC GLA T2R FlashLinearAttention HGRN2
Closest papers by meaning
- Gated Linear Attention Transformers with Hardware-Efficient Training (ICML 2024 · similarity 0.91)
- Forgetting Transformer: Softmax Attention with a Forget Gate (ICLR 2025 · similarity 0.88)
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · similarity 0.86)
- Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.86)
- ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attention (AAAI 2025 · similarity 0.86)
- On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.