AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Tucker Attention: A generalization of approximate attention mechanisms

Timon Klein, Jonas Kusch, Sebastian Sager, Stefan Schnake, Steffen Schotthöfer

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse and structured attention · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use Tucker Attention (low-rank across heads, query, key, value, output) instead of GQA/MLAViT32l and ViT14g fine-tuning · ImageNet-1k top-5 accuracyparameters at comparable accuracyImproved
lower Tucker ranks r2 and r3 rather than head rank r1 (ablation)Tucker Attention on ViT · ImageNet-1ktop-5 accuracyImproved
use Tucker [8,128,64] instead of MHAGPT-2 · OpenWebText zero-shotattention parameters: 56.62 MB -> 10.24 MB; mean accuracy 48.20 -> 46.23Mixed
reduce Tucker key/value ranks from [8,128,128] to [8,64,64] (ablation)Tucker Attention GPT-2 · OpenWebTextvalidation loss: 2.868 -> 2.950Hurt
use shared KV down-projection in MLA (ablation)MLA GPT-2 · OpenWebTexttraining stabilityHurt
add latent RoPE to Tucker Attention (ablation)Tucker Attention GPT-2 · OpenWebTextmean accuracy (%): 47.78 -> 47.33No clear effect
match MLA ranks to GQA nKV=2 (full-rank query, separate KV) (ablation)MLA GPT-2 · OpenWebTextmean accuracy (%): 47.20 -> 47.19No clear effect
use Tucker Attention instead of GQA nKV=8LLaMA3-1B · OpenWebText2validation loss: 2.61 -> 2.65 (Tucker [32,128,128]); parameters 168 MB -> 33.5 MBHurt
use Tucker Attention instead of MLALLaMA3-1B · OpenWebText2validation loss: 2.68 -> 2.65Improved
use Tucker [32,64,64] instead of MHALLaMA3-1B · OpenWebText2training iteration time (ms): 5.19 -> 4.05Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy