Research map / Architectures and efficiency / Attention and transformers
Coneheads: Hierarchy Aware Attention
Albert Tseng, Tao Yu, Toni J.B. Liu, Christopher De Sa
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace dot product attention with penumbral cone attention | transformer_iwslt_de_en · IWSLT'14 De-En | BLEU: 34.56 -> 35.56 | Improved |
| Replace dot product attention with umbral cone attention | DeiT-Ti · ImageNet-1K, 300 epochs | Top-1 accuracy: 72.05 -> 73.14 | Improved |
| Replace dot product attention with umbral cone attention | DeiT-Ti · ImageNet-1K, 500 epochs | Top-1 accuracy: 73.65 -> 74.46 | Improved |
| Replace dot product attention with penumbral cone attention | GAT · PPI | Accuracy: 0.985 -> 0.990 | Improved |
| Use penumbral rather than dot product attention | transformer_lm_wiki103 · WikiText-103, adaptive inputs | Perplexity: 20.86 / 19.22 -> 20.72 / 19.01 | Improved |
| Use umbral rather than dot product attention | transformer_lm_wiki103 · WikiText-103, adaptive inputs | Perplexity: 20.86 / 19.22 -> 21.16 / 19.59 | Hurt |
| Replace dot product attention with cone attention | DiT-B/4 · ImageNet-1K, 400K steps | FID-50K: 68.9 -> 67.7 (penumbral); 68.9 -> 67.6 (umbral) | Improved |
| Use penumbral rather than umbral cones (ablation) | GAT · PPI | Accuracy: 0.989 -> 0.990 | Improved |
| Use umbral rather than penumbral cones (ablation) | GAT · Cora | Accuracy: 0.835 -> 0.836 | Improved |
| Use Exp_O instead of xi mapping (ablation) | Penumbral attention · IWSLT'14 De-En | BLEU: 35.56 -> 35.12 | Hurt |
| Use Exp_O instead of psi mapping (ablation) | Umbral attention · IWSLT'14 De-En | BLEU: 35.07 -> 34.63 | Hurt |
| Reduce token embedding dimension to 16 (ablation) | Cone attention in transformer_iwslt_de_en · IWSLT'14 De-En | BLEU | No clear effect |
Closest papers by meaning
- Mixture-of-Top-$k$ Attention: Efficient Attention as Scalable Fast Weights (NeurIPS 2026 · similarity 0.86)
- Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.86)
- PANDA: Prior-guided Attentional Dual-path Architecture (NeurIPS 2026 · similarity 0.86)
- MoH: Multi-Head Attention as Mixture-of-Head Attention (ICML 2025 · similarity 0.86)
- Elliptical Attention (NeurIPS 2024 · similarity 0.86)
- Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
