Research map / Large language models / Language, safety and interpretability
What Makes Quantization for Large Language Model Hard? An Empirical Study from the Lens of Perturbation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies LLM post-training uniform quantization through a "lens of perturbation" using artificial perturbations, then proposes a simple non-uniform quantization (x^{1/3} transform before uniform quantization) that reduces degradation at W4A16 and W8A8.
Problem
Understanding why zero-shot post-training uniform quantization degrades LLM performance, especially for activations, and how to make LLM quantization more robust.
Approach
View quantization as additive perturbations to weights and activations; artificial perturbation experiments show larger-magnitude values tolerate larger perturbations while smaller values are sensitive, motivating a non-uniform quantizer that applies a nonlinear transform (x^{1/3}) before uniform quantization and its inverse afterward.
Key results
- Non-uniform quantization at W8A8 on LLaMA-7B keeps Wikitext-2 perplexity near full precision (22.28 vs 19.70 full) where uniform W8A8 gives 100.81.
- At W4A16 on LLaMA-7B, non-uniform quantization reaches Wikitext-2 perplexity 20.92 vs 25.05 for RTN uniform and 21.09 for GPTQ, 20.77 for AWQ (AWQ slightly better).
- Positively-correlated (magnitude-aware) perturbations are far less harmful than negatively-correlated ones, which almost completely destroy performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use non-uniform rather than uniform 8-bit weight-and-activation quantization | LLaMA-7B · Wikitext-2 | perplexity: 100.81 -> 22.28 | Helped |
| Use non-uniform rather than uniform 8-bit weight-and-activation quantization | LLaMA-7B · C4 | perplexity: 20.05 -> 7.68 | Helped |
| Use non-uniform rather than uniform 8-bit weight-and-activation quantization | LLaMA-7B · Lambada | accuracy: 86.03 -> 88.52 | Helped |
| Use non-uniform rather than uniform 4-bit weight quantization | LLaMA-7B · Wikitext-2 | perplexity: 25.05 -> 20.92 | Helped |
| Use non-uniform rather than uniform 4-bit weight quantization | LLaMA-7B · C4 | perplexity: 9.12 -> 8.23 | Helped |
| Use non-uniform rather than uniform 4-bit weight quantization | LLaMA-7B · MMLU 4-shot | average score: 33.92 -> 35.83 | Helped |
| Use non-uniform rather than uniform 4-bit weight quantization | LLaMA-7B · Wikitext-2 and C4 | perplexity: Wikitext-2: 20.92 vs 20.77; C4: 8.23 vs 8.03 | Mixed |
| Make perturbation magnitude positively correlated with value magnitude | BLOOM and LLaMA models · Lambada | accuracy | Helped |
| Make perturbation magnitude negatively correlated with value magnitude | BLOOM and LLaMA models · Lambada | accuracy | Hurt |
| Change the perturbation sampling distribution among Gaussian, uniform, and Rademacher | BLOOM and LLaMA models · Lambada | accuracy | No clear effect |
| Clip values above the threshold | LLMs · Lambada, Wikitext-2, and C4 | performance | Hurt |
Surprising findings
- Larger models become easier to quantize under non-uniform quantization, contrary to the trend under uniform quantization where performance deteriorates with scale.
- The sampling distribution of perturbations (Gaussian, uniform, Rademacher) has little impact on performance; magnitude dependence matters most.
Limitations
- Non-uniform transform adds computation, so it offers memory savings but not computation savings in practice.
- Perturbation experiments were limited to BLOOM-560M/3B and LLaMA-7B/13B due to computational cost.
- Simple x^{1/3} transform is chosen for simplicity and not tuned extensively; no efficient hardware kernel.
Experimental setup
Datasets: Lambada Wikitext-2 C4 MMLU
Benchmarks: Lambada Wikitext-2 C4 MMLU
Metrics: accuracy perplexity
Baselines: RTN (uniform quantization) GPTQ AWQ
Builds on: LLM.int8() GPTQ AWQ
Closest papers by meaning
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.92)
- Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models (NeurIPS 2024 · similarity 0.91)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.91)
- Treasures in Discarded Weights for LLM Quantization (AAAI 2025 · similarity 0.91)
- Evaluating Quantized Large Language Models (ICML 2024 · similarity 0.91)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.