Research map / Architectures and efficiency / Attention and transformers
On the Role of Hidden States of Modern Hopfield Network in Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives modern Hopfield attention (MHA), a Transformer self-attention variant that carries a hidden state of attention scores across layers via an exponential moving average, and shows it improves ViT and GPT-2 accuracy/perplexity and mitigates rank collapse without extra parameters.
Problem
Relating modern Hopfield networks to Transformers beyond the adiabatic approximation, and using the resulting hidden-state dynamics to improve Transformer attention, including the rank collapse / token uniformity problem in deep Transformers.
Approach
Keeping the hidden-state dynamics of the modern continuous Hopfield network (instead of the adiabatic limit) yields a new attention layer in which attention logits are an exponential moving average of query-key scores across layers, plus a skip-like term weighted by alpha; this adds no parameters and mitigates rank collapse even without skip connections.
Key results
- On WikiText-103, GPT-2 Medium perplexity improved from 20.85 (self-attention) to 19.61 with MHA (alpha=0.5), and GPT-2 Small from 22.87 to 20.70.
- On ImageNet-1k, ViT-B (86M) accuracy rose from 76.074 (self-attention) to 77.058 with MHA (alpha=0.7), with no added parameters.
- On CIFAR100, ViT-Large MHA (alpha=0.5) reached 75.775 vs 72.910 for self-attention; the paper reports skip-free 12-layer ViT-T collapses to 1% accuracy with self-attention but 39.7% normalized residual norm retained with MHA at depth 12.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace self-attention with MHA (α=0.5) | GPT-2 Small (124M) · WikiText103 | perplexity: 22.87 -> 20.70 | Helped |
| Replace self-attention with MHA (α=0.5) | GPT-2 Medium (350M) · WikiText103 | perplexity: 20.85 -> 19.61 | Helped |
| Replace self-attention with MHA (α=0.5) | LLaMA · WikiText-103, DailyMail, BookCorpus | perplexity: 14.49 -> 14.29; 19.36 -> 18.97; 23.76 -> 23.50 | Helped |
| Replace self-attention with MHA | ViT-Tiny, ViT-Small, ViT-Base, and ViT-Large · CIFAR10 | accuracy: 93.265 -> 93.015; 93.265 -> 93.775; 95.450 -> 95.335; 95.450 -> 95.440; 96.190 -> 96.175; 96.190 -> 96.490; 96.310 -> 96 | Mixed |
| Replace self-attention with MHA | ViT-Tiny, ViT-Small, ViT-Base, and ViT-Large · CIFAR100 | accuracy: 73.080 -> 72.030; 73.080 -> 72.570; 74.485 -> 75.420; 74.485 -> 75.590; 75.360 -> 76.215; 75.360 -> 75.590; 72.910 -> 75 | Mixed |
| Replace self-attention with MHA | ViT-B (86M) · ImageNet-1k | validation accuracy: 76.074 -> 76.434; 76.074 -> 77.058 | Helped |
| Set α to 1 while fixing α'=0.5 (ablation) | ViT-T MHA · CIFAR100 | score: 72.13 -> 1.00 | Hurt |
| Set α to 0 while fixing α'=0.5 (ablation) | ViT-T MHA · CIFAR100 | score: 72.13 -> 69.89 | Hurt |
| Set α' to 0 while fixing α=0.5 (ablation) | ViT-T MHA · CIFAR100 | score: 72.13 -> 71.16 | Hurt |
| Vary α' while fixing α=0.5 (ablation) | ViT-T MHA · CIFAR100 | score: 71.16; 71.13; 72.29; 70.77; 72.12; 72.13; 72.06; 72.02; 71.98; 70.72; 66.10 | Mixed |
| Vary α while fixing α'=0.5 (ablation) | ViT-T MHA · CIFAR100 | score: 69.89; 71.20; 71.26; 71.64; 72.02; 72.13; 72.66; 70.52; 70.46; 67.70; 1.00 | Mixed |
| Increase depth beyond 4 layers in a self-attention network (ablation) | Skip-free ViT-T · CIFAR10 and CIFAR100 | accuracy: CIFAR10: 57.38 -> 48.59 -> 10.00; CIFAR100: 32.25 -> 17.19 -> 1.00 | Hurt |
Surprising findings
- MHA without any skip connection (alpha=0) with nonzero alpha' still mitigates rank collapse, relaxing double-exponential decay to linear decay in theory.
- Setting alpha=1 (removing the value path) drops performance to chance level, while alpha=0 with alpha' fixed still beats the baseline on CIFAR100 in the ablation.
Limitations
- Experiments limited to academic scale: GPT-2 Medium, ViT-L on CIFAR100, and ViT-B on ImageNet-1k.
- Gains on ImageNet-1k are under 1% and results on CIFAR10 are not clearly improved.
- Energy-function interpretation is unresolved for asymmetric (untied) memory matrices.
Experimental setup
Datasets: WikiText-103 CNN DailyMail BookCorpus CIFAR10 CIFAR100 ImageNet-1k Oxford Flowers 102 Food-101 Stanford Dogs Stanford Cars
Benchmarks: WikiText-103 perplexity CNN DailyMail perplexity BookCorpus perplexity CIFAR10 CIFAR100 ImageNet-1k Oxford Flowers 102 Food-101 Stanford Dogs Stanford Cars
Metrics: perplexity classification accuracy cosine similarity distribution normalized residual norm
Baselines: self-attention ViT self-attention GPT-2 self-attention LLaMA
Builds on: modern Hopfield networks Dense Associative Memory self-attention (Vaswani et al.) rank collapse theory (Dong et al.)
Closest papers by meaning
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
- Sparse and Structured Hopfield Networks (ICML 2024 · similarity 0.87)
- Outlier-Efficient Hopfield Layers for Large Transformer-Based Models (ICML 2024 · similarity 0.87)
- What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis (ICLR 2025 · similarity 0.86)
- Learning Linear Attention in Polynomial Time (NeurIPS 2025 · similarity 0.86)
- Nonparametric Modern Hopfield Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.