Research map / Architectures and efficiency / Quantization
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniQuant, a differentiable block-wise post-training quantization method that learns weight clipping (LWC) and equivalent transformations (LET) for LLMs, achieving near-QAT accuracy at PTQ cost.
Problem
Post-training quantization of LLMs relies on hand-crafted quantization parameters, which degrade badly in low-bit settings (W2A16, W4A4), while QAT is too costly for LLMs.
Approach
Freeze full-precision weights and learn a small set of quantization parameters with block-wise error minimization: learnable weight clipping strengths (gamma, beta) for weights and learnable channel-wise scaling/shifting equivalent transformations applied to linear layers and attention, all fusable into the quantized model so no extra parameters or compute are added at inference.
Key results
- On LLaMA-1-13B W2A16, OmniQuant reaches WikiText2 perplexity 13.21 versus GPTQ's 5.5e3 (per-channel setting in the table).
- At W4A4 on LLaMA-1-7B, OmniQuant averages 52.65% zero-shot accuracy versus 48.43% for Outlier Suppression+ and 41.27% for LLM-QAT.
- OmniQuant quantizes LLaMA-2 7B-70B on a single A100-40G within 1-16 hours using 128 samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LWC to LET (ablation) | LLaMA-7B W4A4 · LLaMA-7B W4A4 | average perplexity: 16.97 -> 12.87 | Helped |
| replace LET with SmoothQuant while retaining LWC (ablation) | LLaMA-7B W4A4 · LLaMA-7B W4A4 | average perplexity: 12.87 -> 15.80 | Hurt |
| replace learnable weight clipping with grid-searched weight clipping (ablation) | LET + LWC · LLaMA-7B W4A4 | average perplexity: 12.87 -> 15.82 | Hurt |
| remove LWC (ablation) | OmniQuant · LLaMA-13B W4A4 | perplexity: 10.87 -> 20.75 | Hurt |
| remove LET (ablation) | OmniQuant · LLaMA-13B W4A4 | perplexity: 10.87 -> 5.4e3 | Hurt |
| remove LET (ablation) | OmniQuant · OPT-13B W4A4 | perplexity: 11.65 -> 7.8e3 | Hurt |
| remove channel-wise shifting from LET (ablation) | OmniQuant · LLaMA-13B W4A4 | perplexity: 10.87 -> 11.47 | Hurt |
| remove attention transformation from LET (ablation) | OmniQuant · LLaMA-13B W4A4 | perplexity: 10.87 -> 11.34 | Hurt |
| remove LET from [ln1, (q_proj, k_proj, v_proj)] (ablation) | OmniQuant · LLaMA-7B W4A4 | average accuracy: 52.65 -> 46.79 | Hurt |
| initialize LET scaling as 1 instead of the standard initialization (ablation) | OmniQuant · LLaMA-7B W4A4 | average perplexity: 12.87 -> 13.64 | Hurt |
| quantize Softmax output to 4-bit instead of 16-bit (ablation) | OmniQuant · LLaMA-7B W4A4 | average accuracy: 52.65 -> 48.52 | Hurt |
| train LWC and LET iteratively by iteration instead of simultaneously (ablation) | OmniQuant · LLaMA-7B W4A4 | average accuracy: 52.65 -> 50.91 | Hurt |
Surprising findings
- LET gives negligible benefit for LLaMA weight-only quantization, so it is disabled there, while it is essential for W4A4.
- Learned clipping scales differ strongly by config: per-channel W3A16 gives roughly normal distribution while group-wise settings are mostly minimal clipping.
Limitations
- null
Experimental setup
Datasets: WikiText2
Benchmarks: WikiText2 PTB C4 PIQA ARC BoolQ HellaSwag Winogrande Vicuna-Bench
Metrics: perplexity zero-shot accuracy win rate memory tokens/s
Baselines: RTN GPTQ AWQ SmoothQuant Outlier Suppression+ RPTQ LLM-QAT
Builds on: SmoothQuant Outlier Suppression+ GPTQ AWQ BRECQ
Closest papers by meaning
- OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting (ICLR 2025 · similarity 0.91)
- LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.91)
- OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models (AAAI 2024 · similarity 0.90)
- AffineQuant: Affine Transformation Quantization for Large Language Models (ICLR 2024 · similarity 0.90)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.90)
- ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.