Research map / Architectures and efficiency / RNNs and normalization
Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes U-Hop, a two-stage modern Hopfield retrieval method that first learns a linear kernel feature map by minimizing a separation loss over stored memories, then runs standard kernelized Hopfield retrieval to improve memory capacity and retrieval error.
Problem
Modern Hopfield models suffer from memory confusion (fuzzy retrievals / metastable states) because the inner-product overlap similarity makes stored patterns close in Euclidean geometry, limiting memory capacity and retrieval accuracy.
Approach
Replace the inner-product similarity with a learnable kernel induced by a linear feature map W, trained by minimizing a log average Gaussian-RBF separation loss over the stored memory set (Stage I), then row-normalize W and run kernelized Hopfield retrieval dynamics (Stage II).
Key results
- On CIFAR10, MHM + U-Hop reaches 64.6% max training accuracy and 55.2% test accuracy versus 56.0% and 52.2% for MHM alone.
- On CIFAR100, Sparse MHM + U-Hop reaches 45.4% max training accuracy versus 49.6%... (reported train accuracy for Sparse MHM is 49.6% without U-Hop; U-Hop test 29.0% vs 26.0%)
- Authors report memory retrieval error improves by an average of 30% even with a single Stage I iteration, and learning tasks by an average 3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add U-Hop | Modern Hopfield Model · CIFAR10 image classification | max train accuracy: 56.0% -> 64.6% | Helped |
| add U-Hop | Modern Hopfield Model · CIFAR10 image classification | test accuracy: 52.2% -> 55.2% | Helped |
| add U-Hop | Modern Hopfield Model · CIFAR100 image classification | max train accuracy: 32.3% -> 44.1% | Helped |
| add U-Hop | Modern Hopfield Model · CIFAR100 image classification | test accuracy: 26.3% -> 28.7% | Helped |
| add U-Hop | Modern Hopfield Model · Tiny ImageNet image classification | max train accuracy: 48.9% -> 61.4% | Helped |
| add U-Hop | Modern Hopfield Model · Tiny ImageNet image classification | test accuracy: 12.2% -> 12.7% | Helped |
| add U-Hop | Sparse Modern Hopfield Model · CIFAR10 image classification | max train accuracy: 55.9% -> 66.4% | Helped |
| add U-Hop | Sparse Modern Hopfield Model · CIFAR10 image classification | test accuracy: 52.0% -> 55.4% | Helped |
| add U-Hop | Sparse Modern Hopfield Model · CIFAR100 image classification | max train accuracy: 49.6% -> 45.4% | Hurt |
| add U-Hop | Sparse Modern Hopfield Model · CIFAR100 image classification | test accuracy: 26.0% -> 29.0% | Helped |
| add U-Hop | Sparse Modern Hopfield Model · Tiny ImageNet image classification | max train accuracy: 17.2% -> 60.6% | Helped |
| add U-Hop | Sparse Modern Hopfield Model · Tiny ImageNet image classification | test accuracy: 12.3% -> 12.5% | Helped |
Surprising findings
- Solely increasing embedding dimension does not escape the low-rank bottleneck in attention and Hopfield models, whereas U-Hop's separation learning does.
Limitations
- Separation loss optimality does not guarantee maximal minimal pattern separation R (max-min problem is non-convex).
- Kernelized representation theorem requires M <= d and orthogonality in kernel space, limiting applicability.
- Stage I adds training cost multiplicative in the number of inner iterations.
Experimental setup
Datasets: MNIST CIFAR10 CIFAR100 Tiny ImageNet ETTh1 ETTm1 WTH
Benchmarks: MNIST CIFAR10 CIFAR100 Tiny ImageNet ETTh1 ETTm1 WTH
Metrics: sum-of-square pixel error max training accuracy test accuracy MSE for time series forecasting
Baselines: Modern Hopfield Network Sparse Modern Hopfield Network Dense Associative Memory (10th-order polynomial) L2 distance MHM Manhattan distance MHM STanHop-Net Vision Transformer
Builds on: modern Hopfield networks sparse modern Hopfield networks dense associative memory CCCP Zangwill global convergence
Closest papers by meaning
- On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysis (ICML 2024 · similarity 0.90)
- Nonparametric Modern Hopfield Models (ICML 2025 · similarity 0.90)
- Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codes (NeurIPS 2024 · similarity 0.88)
- Sparse and Structured Hopfield Networks (ICML 2024 · similarity 0.87)
- In-Context Learning as Conditioned Associative Memory Retrieval (ICML 2025 · similarity 0.86)
- Dense Associative Memory with Epanechnikov Energy (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.