Research map / Architectures and efficiency / Attention and transformers

Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse and structured attention · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Exact Flow Linear Attention (EFLA), which replaces the Euler-discretized delta-rule update with the exact closed-form ZOH flow exploiting the rank-1 key outer-product dynamics, and evaluates it on sMNIST robustness, 340M/1.3B language modeling, and MAD.

Problem

Delta-rule linear attention updates are usually viewed as discrete online learning steps, which the paper argues are first-order Euler discretizations of an underlying continuous-time ZOH system; this discretization error may hurt stability under large key norms or corrupted inputs.

Approach

Because the dynamics matrix k k^T is rank-1 (A^2 = lambda A), the matrix exponential and input integral have closed forms, giving an exact-flow delta-rule update with the same algebraic form (replacing beta with alpha = (1-exp(-beta*lambda))/lambda) and compatibility with WY/UT chunkwise parallelism.

Key results

• At 340M/8B tokens, EFLA reduces Wikitext perplexity from 38.09 (DeltaNet) to 35.26 and raises average accuracy from 42.2% to 44.1%.
• At 1.3B/50B tokens, EFLA improves average accuracy from 50.5% (DeltaNet) to 51.8% and reduces LAMBADA perplexity from 17.29 to 16.54.
• On MAD, EFLA raises average score from 65.7% to 66.4% over DeltaNet, with training throughput nearly identical to DeltaNet.

Surprising findings

• Scaling keys by sqrt(d) gave consistently better results for EFLA, while the same scaling on Euler-style DeltaNet can lead to unstable transitions.

Limitations

• Experiments limited to 340M and 1.3B scales; no exhaustive scaling or long-context evaluation.
• Remaining error sources (finite precision, optimization, linear vs softmax gap) not addressed.
• Gains on several benchmarks are small (e.g., 0.1 perplexity on Wikitext at 1.3B, 0.7 MAD average).

Experimental setup

Datasets: SlimPajama (subset) Sequential MNIST MAD synthetic benchmark

Benchmarks: Wikitext LAMBADA PIQA HellaSwag WinoGrande ARC-e ARC-c BoolQ OBQA SciQ Sequential MNIST MAD

Metrics: perplexity accuracy training throughput training loss

Baselines: DeltaNet Gated DeltaNet Mamba-2

Builds on: DeltaNet Gated DeltaNet Mamba-2 WY representation UT transform

Closest papers by meaning

• Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction (NeurIPS 2025 · similarity 0.86)
• Parallelizing Linear Transformers with the Delta Rule over Sequence Length (NeurIPS 2024 · similarity 0.85)
• ZeroS: Zero‑Sum Linear Attention for Efficient Transformers (NeurIPS 2025 · similarity 0.84)
• Attention Mechanism, Max-Affine Partition, and Universal Approximation (NeurIPS 2025 · similarity 0.84)
• Algorithm and Hardness for Dynamic Attention Maintenance in Large Language Models (ICML 2024 · similarity 0.84)
• Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.84)