AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Do Transformers Need Three Projections? Systematic Study of QKV Variants

Ali Kayyam, Anusha Madan Gopal, M Anthony Lewis

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: Block-sparse attention kernels · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
share key and value projections (K=V) instead of separate V (ablation)QKV transformer · SlimPajama (300M)validation perplexity: 5.11 -> 5.27Hurt
share query and key projections (Q=K) instead of separate Q (ablation)QKV transformer · SlimPajama (300M)validation perplexity: 5.11 -> 5.36Hurt
share a single projection for Q, K and V (ablation)QKV transformer · SlimPajama (300M)validation perplexity: 5.11 -> 6.41Hurt
combine K=V with GQA-4 head sharing (Q-GQA-4) (ablation)GQA-4 · SlimPajama (300M)validation perplexity: 5.15 -> 5.32Hurt
share K=V at 1.2B scale (ablation)QKV transformer · SlimPajama (1.2B)perplexity: 5.004 -> 5.128Hurt
add K=V sharing to Q-MQA at 1.2B (ablation)MQA · 5-shot downstream average (1.2B)accuracy (%): 36.37 -> 34.38Hurt
add K=V sharing to GQA-8 (ablation)GQA-8 · 5-shot downstream average (1.2B)accuracy (%): 35.86 -> 36.72Improved
add 2D positional encoding to Q=K≠V (ablation)Q=K≠V · Synthetic tasks averageaccuracy: 0.724 -> 0.761Improved
add 2D positional encoding to Q=K=V (ablation)Q=K=V · Synthetic tasks averageaccuracy: 0.651 -> 0.700Improved
add 2D positional encoding to Q=K≠V (ablation)Q=K≠V · Vision tasks averageaccuracy: 0.766 -> 0.761No clear effect
share Q=K instead of separate Q (ablation)QKV transformer · Vision tasks averageaccuracy: 0.758 -> 0.766No clear effect
share Q=K=V instead of separate projections (ablation)QKV transformer · TinyImageNetaccuracy: 0.331 -> 0.381Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy