Research map / Architectures and efficiency / RNNs and normalization
Dense Associative Memory Through the Lens of Random Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Distributed-representation Dense Associative Memory (DrDAM), approximating the DenseAM energy and dynamics with random Fourier-style features so memories are stored in a fixed-size tensor T, with divergence bounds and empirical validation.
Problem
Dense Associative Memories store each pattern in separate weights, so parameters grow with the number of memories; the paper asks how to get DenseAM storage capacity with a fixed-size, distributed weight representation that still supports recurrent energy-descent retrieval.
Approach
Approximate the DenseAM kernel (exp of negative squared L2 distance / exponential dot product) with random features so that all memories are summed into a single feature-space vector T, enabling energy and gradient computation without storing the original memories; adding a memory costs O(DY) and no new parameters.
Key results
- Retrieval of 20 Tiny ImageNet 64x64x3 images with 40% occlusion succeeds using a distributed tensor of size Y=2e5, about a 20% parameter reduction versus MrDAM (matrix 20x12288).
- Approximation error of DrDAM energy and gradients worsens as inverse temperature beta increases; at beta=50 the energy error near stored patterns is about 10x that at beta=30 and 100x that at beta=10.
- Under step size eta = C2/(L(1+2K beta e^{beta/2})), the trajectory divergence is bounded by O(sqrt(D/Y)) times a constant depending on beta and initial energy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a random-feature distributed memory representation instead of storing each memory separately | DRDAM versus MRDAM · Tiny ImageNet | parameter count: ${\sim}20$\% reduction | Helped |
| Increase the number of random features Y | DRDAM · Random binary patterns; energy and energy gradients | approximation accuracy | Helped |
| Increase inverse temperature β | DRDAM · Queries near stored patterns; energy approximation | energy error: β = 50 has an energy error approximately 10× that of β=30 and 100× that of β=10 | Hurt |
| Increase inverse temperature β and dimension D | DRDAM · Queries far from stored patterns | approximation quality: β=50 approaches 1000× the error of β=10 | Hurt |
| Increase Y when β and D are sufficiently large | DRDAM · Energy and gradient approximation at high β and D | approximation quality | No clear effect |
| Use queries nearer to stored patterns | DRDAM · Memory retrieval | retrieval error | Helped |
| Use SinCos trigonometric random features instead of Positive Random Features (ablation) | DRDAM · Letter | energy and gradient approximation quality | Helped |
| Use SinCos rather than other tested basis functions (ablation) | DRDAM · Letter | energy-gradient approximation quality | Helped |
| Increase the number of stored patterns K while holding Y constant | DRDAM · Approximation error | mean approximation error | Hurt |
| Increase K for random starting positions | DRDAM · Random starting positions | approximation accuracy | Helped |
Surprising findings
- Positive random features (PRF) perform poorly in the high-beta regime, while trigonometric features consistently approximate energy gradients better.
- At large beta and D, increasing the number of random features Y does not improve approximation quality; DrDAM returns near-random gradients and energies.
Limitations
- Approximation quality degrades with high beta, high D, and queries far from stored patterns.
- Limited number of datasets evaluated.
- Does not study compression capability or differing step sizes and layer counts relative to MrDAM; no computational gains over MrDAM claimed.
Experimental setup
Datasets: Tiny ImageNet CIFAR-10 Letter
Benchmarks: CIFAR-10 Tiny ImageNet Letter
Metrics: mean approximation error of energy mean approximation error of gradient normalized Hamming retrieval error compression ratio
Baselines: memory representation DenseAM (MrDAM) Positive Random Features Cos basis Exp basis ExpExp basis
Builds on: Dense Associative Memory random Fourier features positive random features modern Hopfield networks
Closest papers by meaning
- Dense Associative Memory with Epanechnikov Energy (NeurIPS 2025 · similarity 0.89)
- Semantically-correlated memories in a dense associative model (ICML 2024 · similarity 0.88)
- Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codes (NeurIPS 2024 · similarity 0.87)
- Hyperparameter Transfer for Dense Associative Memories (NeurIPS 2026 · similarity 0.87)
- Learning Associative Memories with Gradient Descent (ICML 2024 · similarity 0.86)
- Exponential Dynamic Energy Network for High Capacity Sequence Memory (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.