Research map / Architectures and efficiency / Attention and transformers

Simple linear attention language models balance the recall-throughput tradeoff

ICML 2024 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Based, a hybrid of Taylor-approximated linear attention (global) and small sliding-window softmax attention (local, 64-128 tokens) with IO-aware CUDA kernels, to push the recall-throughput Pareto frontier.

Problem

Attention-based LMs excel at recall but their KV-cache makes generation memory- and throughput-bound, while sub-quadratic alternatives (SSMs, gated convolutions, linear attention) often lag on recall; the paper asks whether a simple architecture can balance recall and throughput.

Approach

Combine second-order Taylor linear attention applied globally (fixed-size recurrent state, feature dimension controls state size) with exact softmax attention in small tensor-core-friendly sliding windows (w<=128), plus gated short convolutions, and write hardware-efficient kernels for linear attention prefill and decoding.

Key results

• Based at 1.3b/50b tokens outperforms Mamba by 10.36 accuracy points on recall-intensive tasks (SWDE, FDA, SQUAD) on average.
• Based achieves up to 24x higher generation throughput than FlashAttention-2 when generating 1024 tokens at batch size 128 with 1.3b parameters on an H100.
• At 1.3b/10b tokens, Based matches Mamba in perplexity (7.43 vs 7.48 on Pile) while being 56% faster than FlashAttention-2 in prefill at 4k length.

Measured effects reported in the paper

Surprising findings

• Linear attention alone struggles on associative recall, but small 64-128 token sliding windows combined with it recover most of full attention's recall (90.8% reported).

Limitations

• Small sliding windows limit long-range modeling, compensated only by the global linear attention layers.
• Taylor feature map requires larger query/key feature dimension (d'=16) that increases parameter count compared to standard attention dimensions.
• Evaluation limited to models up to 1.3b parameters and 50b training tokens.

Experimental setup

Datasets: The Pile MQAR synthetic associative recall SWDE FDA SQUAD LM Eval Harness common sense tasks

Benchmarks: The Pile perplexity MQAR SWDE FDA SQUAD LM Eval Harness

Metrics: perplexity accuracy F1 tokens per ms throughput latency

Baselines: Transformer++ Mamba GLA RWKV v5 H3 Hyena FlashAttention-2 Mistral-7B

Builds on: Mamba FlashAttention-2 Hyena H3 Transformer++

Closest papers by meaning

• Quantifying Memory Utilization with Effective State-Size (ICML 2025 · similarity 0.90)
• Zoology: Measuring and Improving Recall in Efficient Language Models (ICLR 2024 · similarity 0.89)
• Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.89)
• Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.89)
• RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.89)
• RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State (NeurIPS 2026 · similarity 0.89)