Research map / Architectures and efficiency / RNNs and normalization
Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames kernelized modern Hopfield model (KHM) memory storage as an optimal spherical code problem, derives a matching exponential capacity upper/lower bound, and proposes U-Hop+ (projected gradient descent on an average separation loss) to learn the feature map.
Problem
Optimal memorization capacity of kernelized modern Hopfield models (KHMs) and how to choose the feature map to reach it; prior capacity results were lower bounds and lacked optimality guarantees.
Approach
Treat the stored memory set in the learned feature space as a spherical code satisfying a well-separation condition ("memory code"), show optimal capacity corresponds to an optimal spherical code, and use a convex average-separation surrogate loss optimized with projected gradient descent (U-Hop+) that approaches the hardmax objective as temperature tends to zero.
Key results
- On MNIST with U-Hop+, the fraction of queries converging to single-memory metastable states (||p||_0=1) reaches 100% for softmax, 1.5-entmax and sparsemax, versus 3.48%, 69.2% and 88.1% without it.
- U-Hop+ improves test AUC on multiple instance learning tasks, e.g. from 0.876 to 0.921 on one dataset with dense Hopfield pooling.
- The upper capacity bound matches the exponential lower bound in the low-temperature regime, giving M* scaling as c^{D_Phi} for some c>1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply U-Hop+ with Softmax | Modern Hopfield model · Synthetic | metastable state size 1 (%): 0.0 -> 90.0 | Helped |
| apply U-Hop+ with 1.5-entmax | Modern Hopfield model · Synthetic | metastable state size 1 (%): 0.0 -> 100.0 | Helped |
| apply U-Hop+ with Sparsemax | Modern Hopfield model · Synthetic | metastable state size 1 (%): 0.0 -> 100.0 | Helped |
| apply U-Hop+ with Softmax | Modern Hopfield model · MNIST | metastable state size 1 (%): 3.48 -> 100.0 | Helped |
| apply U-Hop+ with 1.5-entmax | Modern Hopfield model · MNIST | metastable state size 1 (%): 69.2 -> 100.0 | Helped |
| apply U-Hop+ with Sparsemax | Modern Hopfield model · MNIST | metastable state size 1 (%): 88.1 -> 100.0 | Helped |
Surprising findings
- Pushing all memories apart in feature space still keeps semantically similar inputs close together on learned features, per the authors' analysis.
- KHMs on MNIST reportedly store all 60,000 patterns using only one-fourth of the dimensions.
Limitations
- Feature map restricted to linear maps Phi(x)=W^T x in the function class H.
- Analysis considers only normalized patterns on the hypersphere, ignoring varying magnitudes.
- Theoretical result is asymptotic in temperature; the authors state tau=1 suffices in practice, so the analysis gives insight rather than practical guidance.
Experimental setup
Datasets: MNIST synthetic Gaussian memories multiple instance learning datasets
Benchmarks: MNIST synthetic memory retrieval multiple instance learning datasets
Metrics: metastable state size test AUC energy landscape basins of attraction
Baselines: softmax modern Hopfield model 1.5-entmax modern Hopfield model sparsemax modern Hopfield model U-Hop HopfieldPooling
Builds on: U-Hop modern Hopfield networks dense associative memory spherical codes
Closest papers by meaning
- On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysis (ICML 2024 · similarity 0.89)
- Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models (ICML 2024 · similarity 0.88)
- Dense Associative Memory Through the Lens of Random Features (NeurIPS 2024 · similarity 0.87)
- Dense Associative Memory with Epanechnikov Energy (NeurIPS 2025 · similarity 0.87)
- Nonparametric Modern Hopfield Models (ICML 2025 · similarity 0.86)
- Scaling Laws for Associative Memories (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.