Research map / Large language models / Reasoning and chain of thought
NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NoMAD-Attention, which replaces query-key multiply-add attention scores with product-quantized 4-bit key codes and SIMD in-register lookups, giving up to 2x decoding speedup for CodeLlama-7B on CPU at 16k context without finetuning.
Problem
LLM inference on CPUs is bottlenecked by the quadratic multiply-add operations in attention score computation, especially at long context lengths.
Approach
Product-quantize keys into 4-bit codes, compute per-query dot-product lookup tables quantized to 8 bits, and keep each sub-quantizer's table in 128-bit SIMD registers so attention scores are computed via batched shuffle lookups instead of MAD operations; works on pre-trained models without finetuning.
Key results
- At 8x key cache compression (d_sub=1, 4-bit codes), NoMAD-Attention perplexity increase is consistently under 4% on WikiText-2 and PTB, while PCA-Attention degrades significantly even at 2x compression.
- NoMAD-Attention-based CodeLlama-7B (4-bit weights) achieves about 2x speedup over the original 4-bit CodeLlama-7B at 16k context length on a 28-core AVX2 Xeon server.
- In an single-thread ablation, NoMAD-Attention achieves 8.3x speedup over MAD-based attention score computation for 16k queries at 16k context.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace dot-product attention with NoMAD-Attention | CodeLlama-7B with 4-bit weights · 16k context | speedup: $2\times$ speedup | Helped |
| Replace MAD-based attention with NoMAD-Attention lookups (ablation) | Single-head attention · 16k queries at 16k context | attention score computation speedup: $8.3\times$ speedup | Helped |
| Use PQ-Attention with 8-bit codes instead of NoMAD-Attention with 4-bit codes (ablation) | Single-head attention · 16k queries at 16k context | attention score computation latency | Hurt |
| Use PQ-Attention with 8-bit codes rather than NoMAD-Attention with 4-bit codes (ablation) | Single-head attention key caching · 16k queries at 16k context | key caching time: $2\times$ more | Hurt |
| Set d_sub=2 instead of d_sub=1 (ablation) | NoMAD-Attention on CodeLlama-7B with 4-bit weights · 16k context | tokens per second: $4$ -> $4.2$ | Helped |
| Use NoMAD-Attention at 8x key cache compression (d_sub=1) | Attention-based LLMs · WikiText-2 test and PTB test | perplexity: consistently less than a 4\% increase | Hurt |
| Apply PCA dimensionality reduction to attention | Attention-based LLMs · WikiText-2 test and PTB test | perplexity | Hurt |
Surprising findings
- Plain product quantization (PQ-Attention) yields only limited speedup over dot-product attention because cache/memory stalls waste cycles, and the lookup table must reside in registers to realize gains.
Limitations
- Evaluated only on AVX2 128-bit SIMD hardware (Xeon E5-2695 V3); results on other CPUs not shown.
- Quality degrades beyond 8x key cache compression, and the 4-bit codebook of 16 centroids constrains accuracy.
- Efficiency evaluated mainly on CodeLlama-7B and a single server; comparisons limited to llama.cpp baselines.
Experimental setup
Datasets: WikiText-2 Penn Treebank (PTB) stack-overflow-questions
Benchmarks: WikiText-2 Penn Treebank (PTB)
Metrics: perplexity prompt processing time decoding latency per token tokens per second attention latency per query
Baselines: dot-product attention (llama.cpp) PCA-Attention PQ-Attention
Builds on: product quantization PCA-Attention llama.cpp FAISS
Closest papers by meaning
- CHAI: Clustered Head Attention for Efficient LLM Inference (ICML 2024 · similarity 0.88)
- Attention-Level Speculation (ICML 2025 · similarity 0.87)
- SparQ Attention: Bandwidth-Efficient LLM Inference (ICML 2024 · similarity 0.87)
- MagicPIG: LSH Sampling for Efficient LLM Generation (ICLR 2025 · similarity 0.87)
- ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization (NeurIPS 2024 · similarity 0.87)
- DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inference (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.