Research map / Architectures and efficiency / Attention and transformers
Quantifying Memory Utilization with Effective State-Size
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes effective state-size (ESS), a rank-based metric of memory utilization for input-varying and input-invariant linear sequence operators, and uses it to analyze, initialize, regularize, distill, and interpret LLM state modulation.
Problem
Existing analyses of causal sequence models either visualize raw operators or report memory capacity (cache/state size), which does not reveal how much of that memory is actually used; a unified quantitative measure is needed across attention, convolutions, and recurrences.
Approach
Define effective state-size as the rank of the causal submatrix H_i = T_{i:,:i-1} of the operator realized by a model (via minimal realization theory), computed per sequence index via SVD using tolerance or entropy (effective rank) variants, and use it to guide distillation, initialization, regularization, and LLM state-modulation analysis.
Key results
- ESS/kv shows a significantly higher correlation with MQAR accuracy than TSS/kv across the entire task-model space.
- In distillation of GLA layers, higher teacher ESS correlates with greater activation distillation loss across all student sizes.
- State modulation hierarchy in 1B models: SA > GLA > WLA > LA, with bigram recall perplexity mirroring this ordering despite similar standard perplexities.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ESS/kv rather than TSS/kv to characterize memory use | Sequence models across the MQAR task-model space · MQAR | correlation with accuracy | Helped |
| Train on a more memory-intensive task with lower TSS | MQAR models · MQAR | ESS-accuracy correlation over training | Helped |
| Use LA rather than GLA or WLA | Recurrent models · MQAR high-kv bucket (kv = $2^7$) | state utilization | Helped |
| Reduce TSS to 8 as task difficulty increases | MQAR models · MQAR | accuracy | Hurt |
| Distill a teacher layer with higher ESS | GLA layers with TSS=256 distilled to smaller students · MQAR | student performance | Hurt |
| Replace the S6 featurizer with GLA featurization | S6-style recurrent model · MQAR | accuracy | Helped |
| Increase the normalization factor α | GLA-S6 · MQAR | accuracy | Helped |
| Add λ||A - I||_F regularization | GLA and WLA · MQAR | accuracy | Helped |
| Remove ESS-based regularization (ablation) | GLA and WLA · MQAR | accuracy | Hurt |
| Increase the normalization factor α | GLA-S6 initialization · MQAR | initialization ESS | Helped |
| Use input-varying state transitions rather than input-invariant transitions | 1B-parameter sequence models · EOS separator-token state modulation | state modulation | Helped |
| Use a model with stronger state modulation | 1B-parameter sequence models · Bigram recall | bigram recall perplexity | Helped |
Surprising findings
- S6 (Mamba without short convolution) fails to learn MQAR at any tested TSS (16-256), with initialization ESS scaling poorly with TSS.
- GLA and WLA trained without the proposed ESS regularizer perform worse than linear attention.
Limitations
- ESS is a function of input and training state, so it is not a fixed architectural property and lacks a guaranteed causal realization for LIVs.
- Entropy-ESS and tolerance-ESS require choosing a tolerance or summary; the tolerance-based variant is somewhat arbitrary.
- Language analysis is qualitative (total ESS plotted against sequence index) and restricted to open-weight models and 1B controlled models.
Experimental setup
Datasets: MQAR selective copying compression FineWeb
Benchmarks: MQAR selective copying compression FineWeb perplexity bigram recall perplexity
Metrics: accuracy ESS/kv correlation perplexity distillation activation loss state utilization (ESS/TSS)
Baselines: linear attention softmax attention GLA WLA S6 (Mamba) TSS / cache size
Builds on: minimal realization theory Eckart-Young-Mirsky theorem effective rank gated linear attention MAD synthetic tasks Mamba
Closest papers by meaning
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.90)
- Proteus: Incremental Memory Activation for Long-Context Sequence Modeling (NeurIPS 2026 · similarity 0.89)
- On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models (NeurIPS 2026 · similarity 0.89)
- Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.88)
- RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State (NeurIPS 2026 · similarity 0.87)
- Understanding and Improving Length Generalization in Recurrent Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.