Research map / Large language models / Language, safety and interpretability
When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies seven linear attention methods on autoregressive LLMs and proposes a causally masked depthwise-conv augmentation with grouped linear attention that is compatible with Medusa-style tree-based speculative decoding.
Problem
Linear attention methods were designed for encoder models and leak future information or fail in autoregressive decoding; they also clash with speculative decoding's tree-based attention, so the combined benefit for training and serving LLMs is unclear.
Approach
Apply a causal (masked) depthwise convolution inside the attention block on top of grouped linear attention with local softmax attention, and unfold the convolution (img2col-style) so it respects Medusa's tree attention mask during speculative decoding.
Key results
- Augmented linear attention reduces LLaMA-2-7B perplexity on PG-19 by 6.67 versus existing linear attention variants (13.47 vs. 21.61 for local-only).
- Combined with Medusa speculative decoding, LLaMA-7B/13B throughput roughly doubles (e.g., 32.3 to 63.3 tokens/s for 7B, 2.0x).
- On the FLASH model, augmented linear attention trains 1.52x and 2.94x faster than original attention at 4K and 8K sequence lengths.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace original attention with augmented linear attention | LLaMA-2-7B · sequence length 8192 | inference latency: 2355.0 -> 1029.7 ms | Helped |
| Replace original attention with augmented linear attention | LLaMA-2-13B · sequence length 8192 | inference latency: 3805.0 -> 1737.1 ms | Helped |
| Add local augmentation and grouped LA (ablation) | LLaMA-2-7B · PG-19 | perplexity: 15.04 -> 13.47 | Helped |
| Add local augmentation and grouped LA (ablation) | LLaMA-2-13B · PG-19 | perplexity: 12.92 -> 11.55 | Helped |
| Add local augmentation to grouped LA (ablation) | GPT-2 · GLUE | average accuracy: 70.22 -> 71.55 | Helped |
| Add local augmentation to grouped LA (ablation) | T5 · GLUE | average accuracy: 62.75 -> 64.51 | Helped |
| Integrate speculative decoding | LLaMA-7B with local attention · MT-Bench | throughput: 32.3 (1.0x) -> 63.3 (2.0x) | Helped |
| Use causal, speculative-decoding-compatible augmentation instead of direct FLASH + Medusa integration | LLaMA-2-7B · speculative decoding | total latency: 137.2 ms -> 49.7 ms (-64%) | Helped |
| Replace original attention with augmented linear attention | FLASH · training sequence length 4K | seconds per training step: 1.60 seconds -> 1.05 seconds | Helped |
| Use augmented linear attention for longer-sequence fine-tuning | LLaMA-2-7B · sequence length extension 4K to 8K | perplexity: 15.29 -> 13.86 | Helped |
Surprising findings
- Advanced encoder-oriented linear attentions (FLASH-Local, YOSO) that are near-baseline on BERT drop significantly on GPT-2 and T5.
- Directly adding a standard DWConv to the value branch in autoregressive LLMs leaks future tokens, yielding zero loss during training but zero evaluation accuracy.
Limitations
- Results for LLaMA are obtained by fine-tuning existing models for only 1K steps rather than pretraining from scratch.
- Accuracy drops on some downstream tasks (e.g., MMLU for 13B) show the augmentation does not uniformly help.
Experimental setup
Datasets: Wiki40B RedPajama GLUE enwik8 PG-19 MT-Bench CNN/Daily Mail Wikitext2 PTB
Benchmarks: GLUE PG-19 Wiki40B MT-Bench BBH PIQA MMLU COPA ARC-C AGNews CNN/Daily Mail
Metrics: accuracy perplexity latency throughput memory ROUGE
Baselines: FLASH-Local FLASH-Global Linformer Performer TransNormer YOSO ReLU linear attention Medusa
Builds on: FLASH Medusa LongLoRA Performer Linformer
Closest papers by meaning
- More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding (NeurIPS 2026 · similarity 0.90)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.89)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.89)
- LoLCATs: On Low-Rank Linearizing of Large Language Models (ICLR 2025 · similarity 0.89)
- Attention Drift: What Auto-Regressive Speculative Decoding Models Learn (NeurIPS 2026 · similarity 0.89)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.