Research map / Architectures and efficiency / Attention and transformers
Understanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Dynamical Systems Framework (DSF), rewriting softmax/linear attention, S6 SSMs, and qLSTM/RG-LRU RNNs as linear time-varying recurrences to compare them, with MQAR, LRA, and WikiText-103 experiments.
Problem
Attention, SSMs, and RNNs are developed in isolation and lack a theoretical common framework explaining their shared principles and differences in expressivity and performance.
Approach
Write attention (separable/kernel), S6, qLSTM, and RG-LRU in one linear structured time-varying recurrence with hidden dimension N = nd, enabling analysis of normalization vs. discretization, state expansion, and softmax as infinite-dimensional state; proposes normalized attention inspired by S6 and a reversed-sigmoid qLSTM transition.
Key results
- On LRA, S6 averages 66.84% versus 53.52% for linear attention, 58.08% for normalized attention, and 55.96% for softmax attention.
- On MQAR (L=512, 64 KV-pairs), softmax attention reaches perfect accuracy while linear attention converges toward it only with larger state expansion n.
- On WikiText-103 (70M params), softmax attention reaches the lowest training perplexity (13.15) versus 16.43 for normalized and 17.42 for linear attention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add input-dependent exponential normalization to linear attention | Linear Attention · LRA | accuracy: 53.52 -> 58.08 | Helped |
| add input-dependent exponential normalization to linear attention | Linear Attention · WikiText-103 | training perplexity: 17.42 -> 16.43 | Helped |
| replace exponential normalization with softplus normalization (ablation) | Normalized Attention · LRA Image | accuracy: 35.96 -> 35.27 | Hurt |
| replace exponential normalization with softplus normalization (ablation) | Normalized Attention · MQAR (L=512, KV-pairs=64) | accuracy: 85.9 -> 84.3 | Hurt |
| replace exponential normalization with sigmoid normalization (ablation) | Normalized Attention · LRA Image | accuracy: 35.96 -> 35.80 | Hurt |
| replace exponential normalization with sigmoid normalization (ablation) | Normalized Attention · MQAR (L=512, KV-pairs=64) | accuracy: 85.9 -> 84.7 | Hurt |
| replace the qLSTM state transition with the S6-inspired reversed-sigmoid state transition (ablation) | qLSTM · MQAR | performance | Helped |
| increase state expansion | Linear Attention · MQAR (L=512, KV-pairs=64) | accuracy | Helped |
Surprising findings
- Normalization (input-dependent, S6-inspired) substantially improves linear attention, closing part of the gap to softmax attention.
- A reversed-sigmoid state transition borrowed from S6 improves qLSTM performance on MQAR.
Limitations
- Insights verified only on synthetic MQAR/LRA and a small WikiText-103 language task.
- DSF-form architectures lack efficient implementations unless existing scan algorithms apply.
- Softmax attention requires infinite state dimension, so only finite approximations are practical.
Experimental setup
Datasets: MQAR LRA WikiText-103
Benchmarks: MQAR LRA WikiText-103
Metrics: accuracy average accuracy training perplexity
Baselines: softmax attention linear attention SSD qLSTM S6
Builds on: Mamba (S6) linear attention qLSTM RG-LRU S4
Closest papers by meaning
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.88)
- Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability (ICLR 2024 · similarity 0.88)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.87)
- Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks (NeurIPS 2025 · similarity 0.87)
- Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers (ICML 2025 · similarity 0.87)
- Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.