Research map / Architectures and efficiency / Attention and transformers
Local Linear Attention: An Optimal Interpolation of Linear and Softmax Attention For Test-Time Regression
Yifei Zuo, Yutong Yin, Zhichen Zeng, Ang Li, Banghua Zhu, Zhaoran Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use local linear attention instead of softmax attention (local constant) | single attention layer, test-time only · piecewise-linear test-time regression (d=64, L=1024) | position-wise MSE | Improved |
| use LLA instead of MesaNet | single attention layer, test-time only · piecewise-linear test-time regression, first segment | position-wise MSE | Hurt |
| add more in-segment data (ablation) | LLA vs softmax attention · piecewise-linear test-time regression | position-wise MSE | Mixed |
| increase data dimension d (ablation) | LLA advantage over baselines · piecewise-linear test-time regression | MSE ratio vs LLA | Improved |
| use LLA instead of Softmax, Mamba, GLA, Hyena, Gated DeltaNet | two-layer models without MLPs · in-context regression on non-stationary data (dx=dy=32) | test error | Improved |
| use LLA instead of softmax and linear baselines | d=128, 2 heads, no short convolution · MQAR associative recall | recall accuracy | Improved |
| use LLA instead of softmax attention | permutation state tracking · permutation state-tracking | test accuracy | No clear effect |
| use blockwise FlashLLA Triton kernel instead of naive implementation | LLA · profiling on H200, d=128 | memory | Improved |
Closest papers by meaning
- Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.90)
- Attention layers provably solve single-location regression (ICLR 2025 · similarity 0.89)
- In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention (ICML 2025 · similarity 0.88)
- Kinetics: Rethinking Test-Time Scaling Law (NeurIPS 2025 · similarity 0.88)
- In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness (NeurIPS 2024 · similarity 0.88)
- STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
