Research map / Architectures and efficiency / Quantization
SqueezeLLM: Dense-and-Sparse Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SqueezeLLM, a post-training weight-only quantization framework combining Fisher-weighted (sensitivity-based) non-uniform k-means quantization with a dense-and-sparse decomposition that keeps outliers and sensitive weights in FP16 sparse format, enabling near-lossless 3/4-bit LLaMA inference with up to 2.4x speedup on A6000.
Problem
LLM generative inference is memory-bandwidth bound, and existing low-bit post-training quantization (uniform, e.g. GPTQ/AWQ) degrades perplexity notably at 3 bits and for smaller models.
Approach
Use non-uniform LUT quantization whose k-means centroids are weighted by a diagonal Fisher information approximation of the final loss, and offload the top-sensitivity and outlier weights (0.45% total) into a full-precision CSR sparse matrix so the dense part has a much narrower range.
Key results
- 3-bit LLaMA-7B SqueezeLLM with 0.45% sparsity reaches C4 perplexity 7.56 vs GPTQ-g128 7.89 and FP16 7.08, at 3.24 avg bits.
- On A6000, 3-bit LLaMA SqueezeLLM achieves up to 2.4x latency speedup over FP16 when generating 128 tokens.
- Dense-only 3-bit SqueezeLLM LLaMA-7B reaches C4 PPL 7.75 vs 9.55 for non-grouped GPTQ and 28.26 for RTN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace uniform quantization with sensitivity-based non-uniform quantization | SqueezeLLM on LLaMA-7B · C4, 3-bit | perplexity: 28.26 -> 7.75 | Helped |
| add 0.45% sparse extraction of sensitive values and outliers (ablation) | SqueezeLLM on LLaMA-7B · C4, 3-bit | perplexity: 7.75 -> 7.56 | Helped |
| remove sensitivity weighting from k-means clustering (ablation) | SqueezeLLM on LLaMA-7B · C4, 3-bit | perplexity: 7.75 -> 18.08 | Hurt |
| add sensitivity-based clustering instead of sensitivity-agnostic clustering (ablation) | SqueezeLLM on LLaMA-7B · C4, 3-bit, 0.05% sparsity | perplexity: 8.10 -> 7.67 | Helped |
| add sensitivity-based clustering instead of sensitivity-agnostic clustering (ablation) | SqueezeLLM on LLaMA-7B · C4, 3-bit, 0.45% sparsity | perplexity: 7.61 -> 7.56 | Helped |
| extract sensitive values as well as outliers into the sparse matrix (ablation) | SqueezeLLM on LLaMA-7B · C4, 3-bit | perplexity | Helped |
| use grouping or grouping plus sparsity instead of Dense-and-Sparse decomposition (ablation) | SqueezeLLM on LLaMA-7B · C4, 3-bit | perplexity | Hurt |
| minimize layer-wise perturbation instead of final-output perturbation (ablation) | SqueezeLLM on LLaMA-7B · C4, 3-bit | perplexity: around 0.3 perplexity points | Hurt |
| increase sparse extraction beyond 0.05% for sensitive values (ablation) | SqueezeLLM on LLaMA-7B · C4, 3-bit | perplexity | No clear effect |
| increase the overall sparse decomposition level (ablation) | SqueezeLLM on LLaMA-2 7B · WikiText2, 4-bit | perplexity: 5.62 -> 5.53 | Helped |
| increase dense-component precision from 3-bit to 4-bit instead of increasing sparsity (ablation) | SqueezeLLM on LLaMA-2 7B · C4 | perplexity: 7.12 vs 7.35 | Helped |
| use a balanced sparse kernel rather than a standard CSR sparse kernel (ablation) | SqueezeLLM on LLaMA-7B · A6000, generating 128 tokens | latency: 3.9 -> 1.7 | Helped |
Surprising findings
- Naive GPTQ with activation ordering and grouping incurs a large latency penalty (0.2x speedup) due to distributed memory accesses, which the dense-and-sparse design avoids.
- Increasing dense bit width generally gives more perplexity gain than adding more sparse outliers beyond about 0.05% sensitive values.
Limitations
- Quantization requires Fisher computation with gradients and k-means clustering that is slower than GPTQ on some models, and peak memory during quantization is higher than GPTQ.
- Speedup depends on custom LUT and balanced sparse CUDA kernels; SpQR speedup comparison relies on reported numbers.
- 2-bit performance is poor for LLaMA2-7B without sparsity (PPL 61.25 on 13B dense-only).
Experimental setup
Datasets: C4 WikiText2 MMLU Vicuna training set
Benchmarks: C4 WikiText2 MMLU Vicuna benchmark
Metrics: perplexity zero-shot accuracy latency (s) peak memory (GB) GPT-4 win/tie/loss
Baselines: RTN GPTQ AWQ SpQR QuIP OmniQuant
Builds on: GPTQ GOBO OBD
Closest papers by meaning
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.92)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.92)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.91)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.91)
- SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models (ICML 2025 · similarity 0.91)
- KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.