Research map / Architectures and efficiency / Attention and transformers
Low-Rank Bottleneck in Multi-head Attention Models
Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank Reddi, Sanjiv Kumar
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase heads from 8 to 32 with the standard head-size heuristic | 24-layer Transformer, embedding size 1024 · SQuAD dev | F1: 90.89$\pm$0.15 -> 90.45$\pm$0.08 | Hurt |
| Increase heads from 8 to 32 with the standard head-size heuristic | 24-layer Transformer, embedding size 1024 · SQuAD dev | EM: 84.1$\pm$0.34 -> 83.48$\pm$0.13 | Hurt |
| Increase heads from 8 to 32 with the standard head-size heuristic | 24-layer Transformer, embedding size 1024 · MNLI dev | accuracy: 85$\pm$0.2 -> 84.4$\pm$0.2 | Hurt |
| Fix head size at 128 while increasing heads from 8 to 32 (ablation) | FixedMultiHead, embedding size 512 · SQuAD dev | F1: 89.6$\pm$0.17 -> 90.95$\pm$0.14 | Improved |
| Fix head size at 128 while increasing heads from 8 to 32 (ablation) | FixedMultiHead, embedding size 512 · SQuAD dev | EM: 82.73$\pm$0.21 -> 84.4$\pm$0.29 | Improved |
| Fix head size at 128 while increasing heads from 8 to 32 (ablation) | FixedMultiHead, embedding size 512 · MNLI dev | accuracy: 83.5$\pm$0.2 -> 84.9$\pm$0.2 | Mixed |
| Increase head size from 32 to 256 (ablation) | FixedMultiHead, embedding size 512, 8 heads · SQuAD dev | F1: 88.53$\pm$0.06 -> 90.33$\pm$0.23 | Improved |
| Increase head size from 32 to 256 (ablation) | FixedMultiHead, embedding size 512, 8 heads · SQuAD dev | EM: 81.19$\pm$0.21 -> 83.36$\pm$0.48 | Improved |
| Increase head size from 32 to 256 (ablation) | FixedMultiHead, embedding size 512, 8 heads · MNLI dev | accuracy: 82.5$\pm$0.1 -> 83.9$\pm$0.2 | Improved |
| Set head size to input sequence length, decoupled from embedding size and head count | Transformer · SQuAD and MNLI dev | performance | Improved |
| Use embedding size 512 with fixed head size instead of the 1024-dimensional baseline | 24-layer Transformer · SQuAD and MNLI dev | performance | Improved |
Closest papers by meaning
- Attention with Trained Embeddings Provably Selects Important Tokens (NeurIPS 2025 · similarity 0.89)
- From Attention to Activation: Unraveling the Enigmas of Large Language Models (ICLR 2025 · similarity 0.88)
- Head Pursuit: Probing Attention Specialization in Multimodal Transformers (NeurIPS 2025 · similarity 0.88)
- Virtual Head Attention (NeurIPS 2026 · similarity 0.88)
- Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.88)
- Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
