Research map / Architectures and efficiency / Attention and transformers
Gated Linear Attention Transformers with Hardware-Efficient Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlashLinearAttention, an I/O-aware chunkwise algorithm for linear attention, and extends it to gated linear attention (GLA) Transformers with data-dependent gates, evaluated on moderate-scale language modeling against Transformer++, RetNet and Mamba.
Problem
Linear attention Transformers underperform softmax attention in language modeling and are slower in wall-clock time than optimized softmax attention because existing chunkwise implementations are not I/O-aware; prior gated variants either use only scalar decays or materialize hidden states in HBM.
Approach
An I/O-aware chunkwise algorithm (FlashLinearAttention) with materialized or non-materialized inter-chunk states, plus a data-dependent low-rank gate G_t = alpha_t^T 1 whose chunkwise form is computed via two-level (secondary) chunking so most intra-chunk matmuls use half-precision tensor cores, and a closed-form gradient for the gates that avoids materializing per-step states.
Key results
- At 1.3B/100B tokens, GLA averages 51.0 on common-sense tasks versus 50.9 for Transformer++, 50.0 for Mamba and 48.9 for RetNet.
- GLA is trained on 2K contexts and extrapolates to sequences beyond 20K without significant perplexity degradation, per the abstract.
- FlashLinearAttention is faster than FlashAttention-2 as a standalone layer even at 1K sequence length, and GLA has higher training throughput than a similarly sized Mamba model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add data-dependent gating rather than use linear attention without a gate (ablation) | GLA Transformer · 340M model, 7B tokens | training perplexity: 23.21 -> 14.77 | Helped |
| use data-dependent fine-grained gates rather than data-independent scalar decay (ablation) | GLA Transformer · 340M model, 7B tokens | training perplexity: 16.55 -> 14.77 | Helped |
| use fine-grained gates rather than data-dependent scalar gates (ablation) | GLA Transformer · 340M model, 7B tokens | training perplexity: 15.56 -> 14.77 | Helped |
| increase the number of heads from 4 to 8 (ablation) | GLA Transformer · 340M model, 7B tokens | training perplexity: 14.77 -> 15.29 | Hurt |
| reduce the number of heads from 4 to 1 (ablation) | GLA Transformer · 340M model, 7B tokens | training perplexity: 14.77 -> 14.61 | Helped |
| replace data-independent decay with data-dependent gates | GLA Transformer versus RetNet · language modeling and zero-shot tasks, 340M and 1.3B | task performance | Helped |
| use GLA instead of Mamba | 1.3B models · SQUAD | score: 35.2 -> 42.6 | Helped |
| use GLA instead of Transformer++ | 1.3B models · HellaSwag | normalized accuracy: 49.6 -> 49.8 | Helped |
| use GLA instead of Transformer++ | 1.3B models · FDA and SWDE | score: 27.4 -> 19.9; 66.6 -> 50.6 | Hurt |
| train GLA on longer contexts rather than 2K contexts | GLA · SlimPajama and PG19 test sets | perplexity | No clear effect |
| use GLA instead of Mamba | 1.3B language models · training sequences beyond 4096 | training throughput | Helped |
Surprising findings
- Reducing GLA to a data-independent scalar decay (RetNet-style) or removing gates degrades perplexity, but a data-dependent scalar gate alone still falls short of the fine-grained gate, showing that fine-grained gating matters.
- Training with TBPTT over 2K segments gives nearly the same perplexity as direct 8K or 24K-length training for GLA.
Limitations
- Experiments limited to 340M and 1.3B models due to compute; scaling behavior beyond 7B is untested
- Gated linear attention is evaluated only on language modeling, not other modalities
- Comparison against Transformer++ and Mamba is at moderate scale; GLA's average advantage over baselines is small
Experimental setup
Datasets: SlimPajama PG19 Wikitext LAMBADA PIQA HellaSwag WinoGrande ARC-easy ARC-challenge FDA SWDE SQuAD
Benchmarks: Wikitext LAMBADA PIQA HellaSwag WinoGrande ARC-easy ARC-challenge FDA SWDE SQuAD MQAR PG19
Metrics: perplexity accuracy acc_norm throughput memory footprint training ppl
Baselines: Transformer++ RetNet Mamba FlashAttention-2 Hyena RWKV-4
Builds on: FlashAttention RetNet Mamba Transformer++ (LLaMA architecture) HGRN
Closest papers by meaning
- Gated Slot Attention for Efficient Linear-Time Sequence Modeling (NeurIPS 2024 · similarity 0.91)
- Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels (NeurIPS 2025 · similarity 0.89)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.88)
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length (NeurIPS 2024 · similarity 0.88)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.88)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.