Research map / Architectures and efficiency / Attention and transformers
Tucker Attention: A generalization of approximate attention mechanisms
Timon Klein, Jonas Kusch, Sebastian Sager, Stefan Schnake, Steffen Schotthöfer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Tucker Attention (low-rank across heads, query, key, value, output) instead of GQA/MLA | ViT32l and ViT14g fine-tuning · ImageNet-1k top-5 accuracy | parameters at comparable accuracy | Improved |
| lower Tucker ranks r2 and r3 rather than head rank r1 (ablation) | Tucker Attention on ViT · ImageNet-1k | top-5 accuracy | Improved |
| use Tucker [8,128,64] instead of MHA | GPT-2 · OpenWebText zero-shot | attention parameters: 56.62 MB -> 10.24 MB; mean accuracy 48.20 -> 46.23 | Mixed |
| reduce Tucker key/value ranks from [8,128,128] to [8,64,64] (ablation) | Tucker Attention GPT-2 · OpenWebText | validation loss: 2.868 -> 2.950 | Hurt |
| use shared KV down-projection in MLA (ablation) | MLA GPT-2 · OpenWebText | training stability | Hurt |
| add latent RoPE to Tucker Attention (ablation) | Tucker Attention GPT-2 · OpenWebText | mean accuracy (%): 47.78 -> 47.33 | No clear effect |
| match MLA ranks to GQA nKV=2 (full-rank query, separate KV) (ablation) | MLA GPT-2 · OpenWebText | mean accuracy (%): 47.20 -> 47.19 | No clear effect |
| use Tucker Attention instead of GQA nKV=8 | LLaMA3-1B · OpenWebText2 | validation loss: 2.61 -> 2.65 (Tucker [32,128,128]); parameters 168 MB -> 33.5 MB | Hurt |
| use Tucker Attention instead of MLA | LLaMA3-1B · OpenWebText2 | validation loss: 2.68 -> 2.65 | Improved |
| use Tucker [32,64,64] instead of MHA | LLaMA3-1B · OpenWebText2 | training iteration time (ms): 5.19 -> 4.05 | Improved |
Closest papers by meaning
- SAS: Simulated Attention Score (NeurIPS 2025 · similarity 0.88)
- Virtual Head Attention (NeurIPS 2026 · similarity 0.87)
- Scale-invariant attention (NeurIPS 2025 · similarity 0.87)
- MoH: Multi-Head Attention as Mixture-of-Head Attention (ICML 2025 · similarity 0.87)
- Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study (ICLR 2025 · similarity 0.86)
- Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
