Research map / Architectures and efficiency / Attention and transformers
Outlier-Efficient Hopfield Layers for Large Transformer-Based Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OutEffHop, an outlier-efficient modern Hopfield model whose retrieval dynamics yield a softmax_1 attention layer that reduces activation outliers (kurtosis, max inf-norm) and improves post-quantization performance across BERT, OPT, ViT, and STanHop-Net.
Problem
Transformer attention softmax forces attention onto low-information "no-op" tokens, creating activation outliers that hurt quantization and efficiency of large transformer models.
Approach
Add a no-op classification dimension and a zero-energy point to the modern Hopfield energy via a refined log-sum-exp; its single-step retrieval dynamics give the softmax_1 attention (exp(z)/(1+sum exp(z))), which lets attention place near-zero weight on all tokens and so suppresses outliers.
Key results
- OutEffHop combined with Clipped Softmax and Gated Attention reduces average kurtosis and maximum infinity norm by roughly 22% and 26% across four models (paper's headline claim).
- On STanHop-Net, OutEffHop ranks first or second in outlier efficiency in 25 of 30 settings, with about 3% lower max infinity norm and 4% lower average kurtosis than the baseline variants.
- On BERT, OutEffHop reduces average kurtosis from 418.7 to 26.6 and max infinity norm from 255.9 to 33.6, while W8A8 perplexity drops from 7.154 to 6.295.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace vanilla attention with OutEffHop | BERT-base-uncased · BERT validation | average kurtosis: 418.724 $\pm$ 0.814 -> 26.564 $\pm$ 0.022 | Helped |
| replace vanilla attention with OutEffHop | BERT-base-uncased · BERT validation | maximum infinity norm: 255.859 $\pm$ 0.004 -> 33.618 $\pm$ 0.000 | Helped |
| replace vanilla attention with OutEffHop | BERT-base-uncased · BERT validation | perplexity (W8A8): 7.154 $\pm$ 0.009 -> 6.295 $\pm$ 0.001 | Helped |
| replace vanilla attention with OutEffHop | OPT-125m · OPT validation | average kurtosis: 23341.513 $\pm$ 27.363 -> 21.542 $\pm$ 0.000 | Helped |
| replace vanilla attention with OutEffHop | OPT-125m · OPT validation | maximum infinity norm: 92.786 $\pm$ 0.002 -> 13.302 $\pm$ 0.001 | Helped |
| replace vanilla attention with OutEffHop | OPT-125m · OPT validation | perplexity (W8A8): 42.012 $\pm$ 19.514 -> 16.429 $\pm$ 0.013 | Helped |
| replace vanilla attention with OutEffHop | ViT-S_16 · ImageNet-1k validation | average kurtosis: 37.104 $\pm$ 0.000 -> 31.601 $\pm$ 0.001 | Helped |
| replace vanilla attention with OutEffHop | ViT-S_16 · ImageNet-1k validation | maximum infinity norm: 272.198 $\pm$ 0.000 -> 249.163 $\pm$ 0.000 | Helped |
| replace vanilla attention with OutEffHop | ViT-S_16 · ImageNet-1k validation | top-1 accuracy (W8A8): 74.935 $\pm$ 0.046 -> 76.313 $\pm$ 0.012 | Helped |
| replace vanilla Hopfield layer with OutEffHop | STanHop-Net · ETTh1 validation | average kurtosis: 2.954 $\pm$ 0.063 -> 2.897 $\pm$ 0.011 | Mixed |
| replace vanilla Hopfield layer with OutEffHop | STanHop-Net · ETTh1 validation | maximum infinity norm: 5.048 $\pm$ 0.232 -> 4.565 $\pm$ 0.209 | Helped |
| combine OutEffHop with Clipped Softmax (ablation) | BERT-base-uncased · BERT validation | average kurtosis: 26.564 $\pm$ 0.022 -> 11.839 $\pm$ 0.001 | Helped |
Surprising findings
- Clipped OutEffHop performs worse than Clipped Softmax on OPT (kurtosis 24127 vs 9731), which the authors tie to Clipped Softmax's known weakness on OPT.
- OutEffHop alone yields only marginal FP16 accuracy changes in some settings and does not address LayerNorm-induced outliers.
Limitations
- Does not address LayerNorm-induced outliers.
- Gains are mixed: Clipped OutEffHop underperforms on OPT, and FP16/W8A8 performance sometimes slightly degrades.
- Requires identifying no-op patterns in the associative-memory setting, though attention-layer use learns this implicitly.
Experimental setup
Datasets: BookCorpus wiki40b/en ImageNet-1k ETTh1 ETTm1 WTH
Benchmarks: ImageNet-1k ETTh1 ETTm1 WTH wiki40b/en BookCorpus
Metrics: average kurtosis maximum infinity norm perplexity top-1 accuracy MSE MAE W8A8 quantized performance
Baselines: Vanilla attention Clipped Softmax Gated Attention Hopfield SparseHopfield STanHop-Net (GSH)
Builds on: modern Hopfield networks softmax_1 attention Clipped Softmax Gated Attention STanHop-Net
Closest papers by meaning
- Nonparametric Modern Hopfield Models (ICML 2025 · similarity 0.88)
- Sparse and Structured Hopfield Networks (ICML 2024 · similarity 0.87)
- On the Role of Hidden States of Modern Hopfield Network in Transformer (NeurIPS 2025 · similarity 0.87)
- Systematic Outliers in Large Language Models (ICLR 2025 · similarity 0.87)
- OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition (ICLR 2025 · similarity 0.86)
- In-Context Denoising with One-Layer Transformers: Connections between Attention and Associative Memory Retrieval (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.