AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

QKV Projections Require a Fraction of Their Memory

Malik Khalaf, Yara Shamshoum, Nitzan Hodos, Yuval Sieradzki, Assaf Schuster

ICLR 2026 · Architectures and efficiency › Attention and transformers · approach: Block-sparse attention kernels · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add PAMM (r=1/512) to Q,K,V projectionsLLaMA-1B pretraining · C4 pretraining (LLaMA-1B)perplexity: 15.56 -> 15.01Improved
add PAMM r=1/128 to LLaMA-60MLLaMA-60M pretraining · C4 pretraining (LLaMA-60M)perplexity: 30.97 -> 31.94Hurt
add PAMM r=1/256 to LLaMA-7BLLaMA-7B pretraining · C4 pretraining (LLaMA-7B, 150K steps)perplexity: 14.61 -> 13.81Improved
add PAMM r=1/512 to LLaMA-7BLLaMA-7B pretraining · C4 pretraining (LLaMA-7B, 150K steps)perplexity: 14.61 -> 13.57Improved
add PAMM to RoBERTa-base QKV (r=1/128)RoBERTa-base GLUE · GLUE average (fine-tuning)average score: 86.28 -> 86.11No clear effect
add PAMM r=1/128 to LoRA fine-tuningPixtral-12B LoRA · AID classification (Pixtral-12B, LoRA)Macro F1: 0.9711 -> 0.9727No clear effect
add PAMM r=1/512 to LoRA fine-tuningPixtral-12B LoRA · AID classification (Pixtral-12B, LoRA)Macro F1: 0.9711 -> 0.9691No clear effect
add PAMM to LLaMA-60M with batch 512, seq 256 (ablation)LLaMA-60M · LLaMA-60M batch 512, seq 256perplexity: 30.97 -> 32.46Hurt
add PAMM to LLaMA-60M with batch 512, seq 128 (ablation)LLaMA-60M · LLaMA-60M batch 512, seq 128perplexity: 37.38 -> 36.43Improved
set epsilon to infinity (drop the neighborhood condition) instead of finite epsilon (ablation)PAMM on LLaMA-60M · C4 pretraining (LLaMA-60M)perplexityImproved
use PAMM generators with cosine-based assignment instead of Uniform-CRS (epsilon=0) (ablation)QKV compression on LLaMA-60M · C4 pretraining (LLaMA-60M)perplexityImproved
reuse generators across steps instead of per-step sampling (ablation)PAMM · LLaMA-1B runtimeperplexityHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy