Research map / Architectures and efficiency / Quantization
Extreme Compression of Large Language Models via Additive Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AQLM, which adapts multi-codebook additive quantization to post-training LLM weight compression with input-aware codebook learning and block-wise fine-tuning, achieving Pareto-optimal 2-3 bit compression of Llama 2 and Mixtral.
Problem
Extreme (2-3 bits per parameter) post-training compression of LLM weights, where existing scalar or lattice quantizers lose substantial accuracy and are not Pareto-optimal against smaller models at the same memory size.
Approach
Extends classic Additive Quantization from nearest-neighbor search to LLM weights by making the objective input-aware (using calibration activations via XX^T), learning codebooks and codes via beam search and Adam, and jointly fine-tuning codebooks and norm parameters per transformer block against the original block outputs.
Key results
- AQLM at 2.02 bits on Llama 2 7B reaches Wiki2 PPL 6.59 and average zero-shot accuracy 57.28, versus 8.22 PPL and 52.23 for QuIP# at 2.02 bits.
- At 2.07 bits on Llama 2 70B, AQLM achieves Wiki2 PPL 3.94 vs 4.16 for QuIP# (2.01 bits), and end-to-end fine-tuning lowers AQLM 2-bit 70B PPL to 3.83.
- AQLM is reported as the first Pareto-optimal scheme below 3 bits per parameter; 2.76-bit 13B outperforms the uncompressed 7B model; GPU inference reaches about 14 tokens/s on Llama 2 70B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AQLM instead of QuIP# at approximately 2 bits | Llama 2 7B · Llama 2 7B, 2-bit quantization | WikiText-2 perplexity: 8.22 -> 6.59 | Helped |
| use AQLM instead of QuIP# at approximately 2 bits | Llama 2 13B · Llama 2 13B, 2-bit quantization | WikiText-2 perplexity: 6.06 -> 5.60 | Helped |
| use AQLM instead of QuIP# at approximately 2 bits | Llama 2 70B · Llama 2 70B, 2-bit quantization | WikiText-2 perplexity: 4.16 -> 3.94 | Helped |
| use AQLM instead of QuIP# at approximately 2 bits | Mixtral 8x7B · Mixtral 8x7B, 2-bit quantization | Average accuracy: 66.34 -> 67.68 | Helped |
| increase calibration data from 128 to 4096 samples (ablation) | AQLM on Llama 2 7B · WikiText-2 | perplexity: 6.994 -> 6.267 | Helped |
| use residual K-means initialization rather than random initialization (ablation) | AQLM on Llama 2 70B q_proj · single linear layer, 3-bit quantization | training iterations to desired loss | Helped |
| fine-tune the whole model end-to-end (ablation) | AQLM · 2-bit quantization | model accuracy | Helped |
| fine-tune only AQ parameters instead of no fine-tuning (ablation) | AQLM · quantized models | model accuracy | Helped |
| fine-tune only RMSNorm scales instead of no fine-tuning (ablation) | AQLM · quantized models | model accuracy | No clear effect |
| vary codebook count and group size at approximately 2 bits (ablation) | AQLM on Llama 2 7B · WikiText-2 | perplexity: 7.6107; 8.1394; 7.3755; 7.8459 | Mixed |
Surprising findings
- Strict 2-bit AQLM on Llama 2 7B is outperformed by 4-bit AQLM on the same model (5.21 vs 5.59 PPL per the paper's comparison), so 2-bit is not Pareto-optimal for the same model.
- Fine-tuning only RMSNorm parameters gives little benefit over no fine-tuning, while codebook fine-tuning drives most of the gain.
Limitations
- Quantization is much more computationally expensive than RTN or GPTQ (about 1 day for 7B on one A100).
- Calibration relies on the RedPajama slice and the baselines were partly adapted by the authors (e.g., QuIP re-implemented for Llama 2).
- Beam search is not necessarily optimal for the code assignment problem; the codebook update uses Adam rather than a closed-form least-squares solver.
Experimental setup
Datasets: RedPajama-v1 WikiText-2 C4 WinoGrande PiQA HellaSwag ARC-easy ARC-challenge
Benchmarks: WikiText-2 C4 WinoGrande PiQA HellaSwag ARC-easy ARC-challenge
Metrics: perplexity zero-shot accuracy average accuracy tokens per second speedup
Baselines: GPTQ SpQR QuIP QuIP# AWQ SqueezeLLM
Builds on: Additive Quantization GPTQ QuIP# Product Quantization
Closest papers by meaning
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.91)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.90)
- BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments (ICLR 2025 · similarity 0.90)
- LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.90)
- PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression (NeurIPS 2024 · similarity 0.90)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.