Research map / Architectures and efficiency / RNNs and normalization

Factual recall in linear associative memories: sharp asymptotics and mechanistic insights

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Layer-wise analysis of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies storage capacity of linear associative memories via a decoupled variant with independent competing outputs per input, using replica-method statistical physics to derive a sharp threshold p log p / d^2 = 1/2 (full rank) and a replica-symmetric prediction for rank-constrained two-layer linear models.

Problem

Determining the exact storage capacity of linear associative memories mapping d-dimensional input embeddings to outputs among many candidates, where strongly correlated constraints (shared outputs) make analysis intractable.

Approach

Introduce a decoupled problem where each input has its own independent set of competing outputs, conjecture it is equivalent in capacity, spectrum and storage mechanism to the original problem, and compute its capacity with the replica method; also extend to rank-constrained two-layer linear networks.

Key results

• Decoupled problem capacity predicted sharply at p log p / d^2 = 1/2 for full-rank linear memory via replica method.
• Empirical accuracy transitions for original and decoupled problems coincide across tested d; Hebbian rule capacity heuristically 1/8.
• Replica-symmetric prediction for rank-constrained memory gives alpha_c approx 0.31, 0.46, 0.50 for kappa = 0.25, 0.5, 1.0.

Surprising findings

• Optimal weights raise target scores just above the extreme-value threshold rather than broadly boosting alignment, beating the Hebbian rule (capacity 1/2 vs 1/8).
• Finite-size corrections of order 1/log p cause empirical thresholds to exceed the asymptotic prediction for the tested dimensions.

Limitations

• Equivalence between original and decoupled problems is conjectured and supported only numerically and heuristically, not proven.
• Replica-symmetric assumption is non-rigorous; for rank-constrained case it gives an upper bound / prediction, and gradient descent may not reach optimum.
• Decoupled-problem simulations limited to d <= 150 due to memory cost.

Experimental setup

Datasets: synthetic Gaussian input and output embeddings

Benchmarks: synthetic associative memory tasks (original and decoupled problems)

Metrics: retrieval accuracy cross-entropy loss singular value distribution critical load alpha_c

Baselines: Hebbian weight matrix

Builds on: Hebbian ansatz (Cabannes et al. associative memory scaling) replica method Gaussian equivalence

Closest papers by meaning

• Scaling Laws for Associative Memories (ICLR 2024 · similarity 0.90)
• On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashing (NeurIPS 2026 · similarity 0.89)
• Semantically-correlated memories in a dense associative model (ICML 2024 · similarity 0.89)
• Understanding Factual Recall in Transformers via Associative Memories (ICLR 2025 · similarity 0.88)
• Learning Associative Memories with Gradient Descent (ICML 2024 · similarity 0.88)
• Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory (NeurIPS 2026 · similarity 0.88)