Research map / Architectures and efficiency / RNNs and normalization
Dense Associative Memory with Epanechnikov Energy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes log-sum-ReLU (LSR), a DenseAM energy based on the Epanechnikov kernel, which exactly retrieves exponentially many memories while creating many emergent local minima (centroids of overlapping basins), with log-likelihood comparable to LSE.
Problem
Existing log-sum-exp (LSE) dense associative memories either memorize exactly or generate novel patterns, but not both; LSE cannot have more local minima than stored patterns while retaining perfect recall.
Approach
Replace the exponential separation function of LSE with a shifted ReLU (Epanechnikov kernel, optimal for KDE in MISE), giving compactly supported basins; exact retrieval of all memories is preserved while overlapping basins create emergent local minima at subset centroids.
Key results
- LSR produces orders of magnitude more emergent memories than stored patterns at critical beta while preserving most stored patterns (>60%).
- LSR memories match LSE in average log-likelihood on mixture-of-Gaussians density while producing more diverse samples (500 LSE queries converge to ~10 memories).
- With the same beta on VAE latents, LSR generates an order of magnitude more total memories than LSE, which can retrieve at most M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LSE energy with LSR energy | DenseAM on a mixture of Gaussians · Mixture of Gaussians | average log-likelihood: match LSE in log-likelihood | No clear effect |
| Replace LSE energy with LSR energy | DenseAM on a mixture of Gaussians · Mixture of Gaussians | unique samples | Helped |
| Replace LSE energy with LSR energy | DenseAM on a mixture of Gaussians · Mixture of Gaussians | original memories recoverable | Helped |
| Use LSR energy instead of LSE at the same β | DenseAM in latent image space · MNIST and TinyImageNet latent spaces | total memories: order of magnitude more total memories | Helped |
| Use LSR energy | DenseAM in latent image space · TinyImageNet latent space | original patterns recoverable | Helped |
| Replace Epanechnikov kernel with TriWeight kernel (ablation) | DenseAM energy · Kernel comparison | emergent memories | Hurt |
| Replace Epanechnikov kernel with Quartic kernel (ablation) | DenseAM energy · Kernel comparison | emergent memories | Hurt |
| Replace Epanechnikov kernel with Cosine kernel (ablation) | DenseAM energy · Kernel comparison | emergent memories | No clear effect |
Surprising findings
- Emergent memories can appear even with just 2 stored patterns of any dimension.
- The triweight kernel, despite compact support, did not produce emergence at any tested temperature.
Limitations
- Gradient vanishes outside the support of stored patterns, making gradient-based retrieval ineffective for distant queries.
- Emergent memories occur only at centroids of overlapping basins, limiting creativity to a predictable subset of the convex hull.
- Emergent memories in Tiny ImageNet latents are often blurry and undesirable for high-fidelity generation.
Experimental setup
Datasets: MNIST TinyImageNet synthetic mixture of Gaussians
Benchmarks: MNIST TinyImageNet synthetic mixture of Gaussians log-likelihood
Metrics: number of local minima / emergent memories average log-likelihood number of unique samples number of original memories recovered
Baselines: log-sum-exp energy Gaussian kernel KDE
Builds on: dense associative memory modern Hopfield networks kernel density estimation
Closest papers by meaning
- Dense Associative Memory Through the Lens of Random Features (NeurIPS 2024 · similarity 0.89)
- Semantically-correlated memories in a dense associative model (ICML 2024 · similarity 0.87)
- Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codes (NeurIPS 2024 · similarity 0.87)
- Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models (ICML 2024 · similarity 0.86)
- Hyperparameter Transfer for Dense Associative Memories (NeurIPS 2026 · similarity 0.85)
- Learning Associative Memories with Gradient Descent (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.