Research map / Architectures and efficiency / Attention and transformers
Are Sixteen Heads Really Better than One?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove individual attention heads after training (ablation) | WMT large Transformer and BERT base-uncased · newstest2013; MNLI-matched validation | BLEU; accuracy | No clear effect |
| reduce many individual layers to one attention head (ablation) | WMT large Transformer and BERT base-uncased · newstest2013; MNLI-matched validation | BLEU; accuracy: None of these results are statistically significant with p<0.01 | No clear effect |
| reduce layer 6 of encoder-decoder attention to one head (ablation) | WMT large Transformer · newstest2013 | BLEU: -13.56 | Hurt |
| prune heads iteratively in increasing importance-score order (ablation) | WMT large Transformer · newstest2013 | BLEU: 20% | No clear effect |
| prune heads iteratively in increasing importance-score order (ablation) | BERT base-uncased · MultiNLI-matched validation | accuracy: 40% | No clear effect |
| use importance score I_h rather than individual oracle performance difference to order pruning (ablation) | WMT large Transformer and BERT base-uncased · newstest2013; MultiNLI-matched validation | BLEU; accuracy | Helped |
| prune 50% of attention heads (ablation) | BERT base-uncased · BERT inference, batch size 16 | inference speed: 114.0\pm 3.6 -> 134.0\pm 3.6; (+17.5\%) | Helped |
| prune heads from encoder-decoder attention rather than self-attention (ablation) | WMT large Transformer · WMT translation | BLEU: more than 60% | Hurt |
| prune heads from encoder or decoder self-attention (ablation) | WMT large Transformer · WMT translation | BLEU: 20%; around 30 | No clear effect |
| prune heads based on importance during early training rather than after heads have differentiated (ablation) | IWSLT German-to-English model · newstest2013 | relative BLEU degradation: epochs 1 and 2; epoch 10 onwards; 40% | Mixed |
Closest papers by meaning
- MoH: Multi-Head Attention as Mixture-of-Head Attention (ICML 2025 · similarity 0.85)
- Head Pursuit: Probing Attention Specialization in Multimodal Transformers (NeurIPS 2025 · similarity 0.85)
- On the Role of Attention Heads in Large Language Model Safety (ICLR 2025 · similarity 0.84)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.84)
- Virtual Head Attention (NeurIPS 2026 · similarity 0.84)
- Heads up! Large Language Models Can Perform Tasks Without Your Instruction via Selective Attention Head Masking (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.