Research map / Architectures and efficiency / Attention and transformers
QKV Projections Require a Fraction of Their Memory
Malik Khalaf, Yara Shamshoum, Nitzan Hodos, Yuval Sieradzki, Assaf Schuster
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PAMM (r=1/512) to Q,K,V projections | LLaMA-1B pretraining · C4 pretraining (LLaMA-1B) | perplexity: 15.56 -> 15.01 | Improved |
| add PAMM r=1/128 to LLaMA-60M | LLaMA-60M pretraining · C4 pretraining (LLaMA-60M) | perplexity: 30.97 -> 31.94 | Hurt |
| add PAMM r=1/256 to LLaMA-7B | LLaMA-7B pretraining · C4 pretraining (LLaMA-7B, 150K steps) | perplexity: 14.61 -> 13.81 | Improved |
| add PAMM r=1/512 to LLaMA-7B | LLaMA-7B pretraining · C4 pretraining (LLaMA-7B, 150K steps) | perplexity: 14.61 -> 13.57 | Improved |
| add PAMM to RoBERTa-base QKV (r=1/128) | RoBERTa-base GLUE · GLUE average (fine-tuning) | average score: 86.28 -> 86.11 | No clear effect |
| add PAMM r=1/128 to LoRA fine-tuning | Pixtral-12B LoRA · AID classification (Pixtral-12B, LoRA) | Macro F1: 0.9711 -> 0.9727 | No clear effect |
| add PAMM r=1/512 to LoRA fine-tuning | Pixtral-12B LoRA · AID classification (Pixtral-12B, LoRA) | Macro F1: 0.9711 -> 0.9691 | No clear effect |
| add PAMM to LLaMA-60M with batch 512, seq 256 (ablation) | LLaMA-60M · LLaMA-60M batch 512, seq 256 | perplexity: 30.97 -> 32.46 | Hurt |
| add PAMM to LLaMA-60M with batch 512, seq 128 (ablation) | LLaMA-60M · LLaMA-60M batch 512, seq 128 | perplexity: 37.38 -> 36.43 | Improved |
| set epsilon to infinity (drop the neighborhood condition) instead of finite epsilon (ablation) | PAMM on LLaMA-60M · C4 pretraining (LLaMA-60M) | perplexity | Improved |
| use PAMM generators with cosine-based assignment instead of Uniform-CRS (epsilon=0) (ablation) | QKV compression on LLaMA-60M · C4 pretraining (LLaMA-60M) | perplexity | Improved |
| reuse generators across steps instead of per-step sampling (ablation) | PAMM · LLaMA-1B runtime | perplexity | Hurt |
Closest papers by meaning
- Memory-Efficient LLM Training with Online Subspace Descent (NeurIPS 2024 · similarity 0.85)
- SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization (ICML 2025 · similarity 0.85)
- Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks (NeurIPS 2025 · similarity 0.85)
- Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples (NeurIPS 2025 · similarity 0.84)
- CommVQ: Commutative Vector Quantization for KV Cache Compression (ICML 2025 · similarity 0.84)
- SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
