Research map / Architectures and efficiency / Attention and transformers
Push, Pop, Parallelize: Stack-Augmented Linear Attention via the Delta Rule
Anh T Nguyen, Saleh Momeni, Ashutosh Chaubey, Changnan Xiao, Bing Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a differentiable stack to DeltaNet | DeltaNet · Formal languages (average over 8 tasks) | accuracy: 65.5 -> 79.4 | Improved |
| add a differentiable stack to DeltaNet | DeltaNet · Formal languages: Stack Manipulation | accuracy: 56.6 -> 98.8 | Improved |
| add a differentiable stack to DeltaNet | DeltaNet · Formal languages: Parity | accuracy: 64.3 -> 96.8 | Improved |
| add a differentiable stack to DeltaNet | DeltaNet · Formal languages: Dyck Mod Arith. w/ Brackets | accuracy: 33.2 -> 35.2 | No clear effect |
| add a differentiable stack to DeltaNet | DeltaNet · Formal languages: Cycle Navigation | accuracy: 46.7 -> 46.4 | No clear effect |
| add a differentiable stack to DeltaNet | DeltaNet · MAD benchmark (average) | average score: 71.8 -> 72.2 | No clear effect |
| add a differentiable stack to DeltaNet (s = 8, s = 16) | DeltaNet · FineWeb-Edu LM, 340M params / 15B tokens | perplexity and zero-shot average: 45.5 -> 45.9 (s=8), 46.2 (s=16) avg acc | No clear effect |
| add a differentiable stack to DeltaNet (s = 8) | DeltaNet · FineWeb-Edu LM, 760M params / 30B tokens | average zero-shot accuracy: 51.4 -> 52.0 | Improved |
| add a differentiable stack to DeltaNet | DeltaNet 340M · CodeParrot and OpenThoughts-114k-Math length extrapolation to 16k | per-token loss | Improved |
| increase stack size from 8 to 48 (ablation) | DeltaStack · Formal languages: Parity, Reverse String, Stack Manipulation | accuracy | Improved |
| increase stack size (ablation) | DeltaStack · Formal languages: state tracking (Parity) | accuracy | No clear effect |
| add a differentiable stack to DeltaNet | DeltaNet 1.3B · Peak training memory, A100 | peak memory (GiB): 41.3 -> 45.1 | Hurt |
Closest papers by meaning
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length (NeurIPS 2024 · similarity 0.91)
- Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025 · similarity 0.89)
- Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity (NeurIPS 2026 · similarity 0.88)
- Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics (NeurIPS 2026 · similarity 0.87)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.87)
- ZeCO: Zero-Communication Overhead Sequence Parallelism for Linear Attention (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
