Research map / Architectures and efficiency / RNNs and normalization
Sparse and Structured Hopfield Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Unifies sparse modern Hopfield networks with Fenchel-Young losses (Hopfield-Fenchel-Young energies), proves exact one-step retrieval under margin conditions, and extends to structured retrieval of pattern associations via SparseMAP.
Problem
Modern Hopfield networks (and transformer attention-based updates) use softmax, which prevents exact retrieval and can yield metastable mixtures; sparse variants retrieve only approximately, and retrieval of pattern associations is not addressed.
Approach
Define Hopfield energies as a convex-concave decomposition where the concave part is a negative Fenchel-Young loss for a generalized negentropy (Tsallis or norm), so update rules are sparse transformations (entmax, normmax, sparsemax); margin properties of the loss give exact retrieval with exponential capacity, and SparseMAP over structured sets enables retrieval of pattern associations.
Key results
- On MNIST metastable-state analysis, 1.5-entmax and 2-entmax and 5-normmax show few metastable states at beta=1, while k-subsets retrieves k-pattern associations as expected.
- SparseMAP k=2 reaches 97.7% on 2-MIL MNIST; SparseMAP pooling beats sparse pooling variants on 2 of 3 MIL benchmarks.
- Sequential k-subsets rationalizer obtains HRO F1 0.63 on BeerAdvocate versus 0.61 for SPECTRA, with comparable downstream performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SparseMAP k-subsets with k=K instead of other pooling variants | Hopfield pooling for MNIST K-MIL · MNIST K=2 and K=3 | accuracy: 97.7\pm0.3; 96.5\pm0.5 | Helped |
| Increase entmax alpha from 1 to 2 (ablation) | Hopfield pooling · MNIST K=2 | accuracy: 94.6\pm0.5 -> 96.7\pm0.5 | Helped |
| Increase normmax alpha from 2 to 5 (ablation) | Hopfield pooling · MNIST K=3 | accuracy: 93.9\pm0.7 -> 95.8\pm0.4 | Helped |
| Use 1-entmax instead of other transformations | Hopfield pooling · MNIST K=1 | accuracy: 98.4\pm0.2 | Helped |
| Use SparseMAP pooling instead of sparse pooling | Hopfield pooling · Fox and Elephant MIL benchmarks | ROC AUC: 66.1\pm0.6 -> 67.4\pm2.0; 92.4\pm0.7 -> 93.2\pm0.7 | Mixed |
| Use sequential k-subsets rather than non-sequential k-subsets (ablation) | SparseMAP text rationalizer · Beer HRO | F1: .42 { (.29/.62)} -> .63 { (.49/.70)} | Helped |
| Use sequential k-subsets rather than non-sequential k-subsets (ablation) | SparseMAP text rationalizer · Beer MSE | MSE: .017 { (.017/.018)} -> .020 { (.018/.021)} | Hurt |
| Increase entmax alpha (ablation) | Sparse Hopfield network · MNIST test queries, beta=1 | single-pattern metastable state frequency (%): 97.8 -> 100.0 | Helped |
| Use sequential structure to encourage adjacent selected tokens | SparseMAP text rationalizer · Beer HRO | F1: .61 -> .63 | Helped |
Surprising findings
- 1-entmax (softmax) is best on MNIST K=1 (98.4%) despite the theory favoring sparse retrieval
- normmax tends to stabilize in small persistent metastable states as alpha increases
Limitations
- Experiments are small-scale benchmarks without large-scale transformer evaluation.
- Normmax shows persistent metastable states at larger alpha.
Experimental setup
Datasets: MNIST Elephant Fox Tiger SST AgNews IMDB BeerAdvocate
Benchmarks: MNIST Elephant Fox Tiger SST AgNews IMDB BeerAdvocate
Metrics: accuracy ROC AUC F1 MSE metastable state distribution
Baselines: softmax Hopfield pooling (1-entmax) sparsemax Hopfield pooling (2-entmax) HardKuma SPECTRA
Builds on: modern Hopfield networks (Ramsauer et al.) sparse Hopfield networks (Martins et al. / sparsemax Hopfield) Fenchel-Young losses entmax SparseMAP
Closest papers by meaning
- Nonparametric Modern Hopfield Models (ICML 2025 · similarity 0.90)
- Outlier-Efficient Hopfield Layers for Large Transformer-Based Models (ICML 2024 · similarity 0.87)
- On the Role of Hidden States of Modern Hopfield Network in Transformer (NeurIPS 2025 · similarity 0.87)
- Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models (ICML 2024 · similarity 0.87)
- On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysis (ICML 2024 · similarity 0.87)
- In-Context Denoising with One-Layer Transformers: Connections between Attention and Associative Memory Retrieval (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.