Research map / Architectures and efficiency / Quantization
Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic study of post-training quantization (RTN, GPTQ, ZeroQuant variants, fine-grained quantization) on OPT and BLOOM (125M to 176B) across weight-only and weight-and-activation settings, plus LoRC, a low-rank SVD compensation of the quantization error.
Problem
Post-training quantization of LLMs lacks a systematic study across quantization schemes, model families, and bit precisions, and existing PTQ methods may not reach full-precision quality at INT4 weights or W4A8.
Approach
Apply a low-rank SVD approximation to the weight quantization error matrix (LoRC) with 8-bit low-rank factors, added on top of GPTQ/ZeroQuant and fine-grained quantization, to recover near-FP16 quality with minimal extra parameters.
Key results
- LoRC on GPTQ with W4A16 (256 block size) brings OPT-66B perplexity to 10.29 vs 10.33 FP16 and BLOOM-176B to 10.93 vs 10.90 FP16 (Table 7).
- At 2-bit weights, LoRC reduces OPT-6.7b perplexity from 120.56 (GPTQ, per-row) to 24.17 with LoRC, and with 256 block size from 23.13 to 16.27.
- Without fine-grained quantization, no tested optimization method reaches Class-1 degradation for INT4 weight-only or W4A8 except GPTQ on OPT-30B W4A16; models >=13B reach Class-1 with fine-grained 4-bit weights.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add low-rank compensation to GPTQ W4A16 | OPT-6.7B · OPT-6.7B average PPL | PPL: 12.28 -> 12.10 | Helped |
| Add low-rank compensation to GPTQ W4A16 | OPT-13B · OPT-13B average PPL | PPL: 11.42 -> 11.36 | Helped |
| Add low-rank compensation to GPTQ W4A16 | OPT-30B · OPT-30B average PPL | PPL: 10.78 -> 10.76 | Helped |
| Add low-rank compensation to GPTQ W4A16 | OPT-66B · OPT-66B average PPL | PPL: 10.78 -> 10.34 | Helped |
| Add low-rank compensation to GPTQ W4A16 | BLOOM-176B · BLOOM-176B average PPL | PPL: 11.02 -> 10.98 | Helped |
| Add low-rank compensation to GPTQ W3A16 | OPT-6.7B · OPT-6.7B average PPL | PPL: 12.79 -> 12.40 | Helped |
| Add low-rank compensation to GPTQ W2A16 | OPT-30B · OPT-30B average PPL | PPL: 25.74 -> 14.39 | Helped |
| Use block-size-256 fine-grained weight quantization instead of per-row quantization (ablation) | OPT W4A16 · OPT-6.7B average PPL | PPL: 12.10 -> 12.05 | Helped |
| Use block-size-256 fine-grained weight quantization instead of per-row quantization (ablation) | OPT W4A16 · OPT-13B average PPL | PPL: 11.36 -> 11.28 | Helped |
| Use block-size-128 activation quantization instead of per-row activation quantization (ablation) | BLOOM-176B W4A8 · BLOOM-176B average PPL | PPL: 11.19 -> 11.03 | Helped |
| Use block-size-128 weight and activation quantization instead of per-row weight and activation quantization (ablation) | BLOOM-176B W4A8 · BLOOM-176B average PPL | PPL: 11.19 -> 10.96 | Helped |
| Represent LoRC matrices with INT8 instead of FP16 (ablation) | OPT W4A16 + LoRC · OPT-6.7B average PPL | PPL: 12.08 -> 12.10 | No clear effect |
Surprising findings
- Smaller models often outperform larger models under INT8 activation quantization, and OPT models diverge from 6.7B onward under activation quantization, attributed to poorly trained LayerNorm weights.
- BLOOM-176B shows only small degradation under INT8 activation quantization while OPT-30B and -66B degrade badly.
Limitations
- Results are limited to OPT and BLOOM model families and perplexity metrics on language modeling datasets.
- Even with best methods, INT4 and W4A8 quantization of smaller models (<=10B) remain Class-2 or Class-3 degradation.
- Activation quantization for OPT models beyond 6.7B diverges, and the paper does not fully resolve this.
Experimental setup
Datasets: WikiText-2 PTB C4
Benchmarks: WikiText-2 PTB C4
Metrics: perplexity perplexity degradation
Baselines: RTN GPTQ ZQ-Local ZQ-Global SmoothQuant
Builds on: GPTQ ZeroQuant SmoothQuant OBQ
Closest papers by meaning
- Evaluating Quantized Large Language Models (ICML 2024 · similarity 0.95)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.93)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.92)
- ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals (ICML 2025 · similarity 0.92)
- CBQ: Cross-Block Quantization for Large Language Models (ICLR 2025 · similarity 0.92)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.