Research map / Architectures and efficiency / Quantization
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SpQR, a hybrid sparse-quantized weight format and GPTQ-based post-training quantization that isolates ~1% outlier weights in fp16 and uses tiny groups with bilevel-quantized statistics, achieving near-lossless (<1% perplexity increase) 3-4 bit LLM compression.
Problem
Standard 3-4 bit weight quantization of LLMs causes significant accuracy degradation, especially for small (1-10B) models suited to edge deployment, and existing formats do not isolate the sensitive weights that cause large errors.
Approach
Identify weights whose quantization induces large layer-output error (via a GPTQ-derived sensitivity criterion) and store them in 16-bit in a CSR-like sparse format, while quantizing the rest with very small groups (16 weights) whose scales and zero-points are themselves quantized to 3 bits in a second level.
Key results
- SpQR reaches within 1% perplexity of the 16-bit baseline at about 4.6-4.71 bits per parameter for LLaMA models.
- At 4 bits per parameter, SpQR roughly halves the error relative to the 16-bit baseline compared to GPTQ.
- An optimized sparse-plus-quantized GPU kernel makes SpQR 20-30% faster than 16-bit inference for token generation on an A100 (e.g., 57 vs 47 tokens/s for LLaMA-7B from scratch).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SpQR instead of GPTQ or round-to-nearest at similar model size | LLM weight quantization · LLaMA WikiText2 and zero-shot tasks | perplexity and accuracy | Helped |
| Use 4-bit SpQR instead of 4-bit GPTQ | LLM weight quantization · LLaMA and Falcon perplexity benchmarks | perplexity error relative to 16-bit | Helped |
| Use the optimized sparse matrix multiplication algorithm instead of PyTorch sparse multiplication | SpQR inference · LLaMA token generation | tokens/s: about 20-30% | Helped |
| Use small weight groups with quantized statistics instead of larger groups with 16-bit statistics (ablation) | 3-bit SpQR without outliers · Language modeling benchmarks | language modeling loss | Helped |
| Use unstructured outliers instead of row outliers or column outliers (ablation) | 3-bit SpQR · LLaMA-65B perplexity | perplexity | Helped |
| Round zero-points to 3-bit integers (ablation) | SpQR · Quantization evaluation | perplexity | Hurt |
| Remove the activation-order heuristic (ablation) | SpQR · Quantization evaluation | language modeling loss | Hurt |
Surprising findings
- Standard PyTorch sparse matrix multiplication is not faster than 16-bit inference, so a custom sparse kernel was required for speedups.
- Quantizing the group statistics (scales and zero-points) to 3 bits improves perplexity over 16-bit statistics at similar memory footprint.
Limitations
- Only perplexity and zero-shot accuracy evaluated; generative quality not measured.
- Sparse matmul not fused with dense quantized matmul, leaving inference speed gains on the table.
- Outlier storage costs ~32 bits per outlier, adding overhead; grouping outliers left for future work.
Experimental setup
Datasets: RedPajama RefinedWeb C4 WikiText2 Penn Treebank
Benchmarks: WikiText2 Penn Treebank C4 WinoGrande PiQA HellaSwag ARC-easy ARC-challenge
Metrics: perplexity zero-shot accuracy tokens per second average bits per parameter
Baselines: GPTQ round-to-nearest (RTN) LLM.int8() outlier criterion
Builds on: GPTQ Optimal Brain Surgeon LLM.int8() SparseGPT
Closest papers by meaning
- SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression (ICML 2025 · similarity 0.93)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.93)
- SqueezeLLM: Dense-and-Sparse Quantization (ICML 2024 · similarity 0.92)
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs (NeurIPS 2025 · similarity 0.92)
- LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.91)
- 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.