Research map / Architectures and efficiency / Quantization
LQER: Low-Rank Quantization Error Reconstruction for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LQER and L²QER, a post-training quantization method that adds a low-rank, high-precision correction (SVD of an activation-scaled quantization error) to a low-bit MXINT weight, achieving near-lossless W4A8 LLM quantization without iterative optimization.
Problem
Post-training quantization of LLMs to low bitwidth (especially weight-activation W4A8) loses accuracy due to outliers, while fine-grained outlier handling methods incur high optimization cost or irregular scatter/gather computation patterns in hardware.
Approach
Approximate the weight as a low-precision full-rank quantized matrix plus a high-precision low-rank correction obtained by SVD of the quantization error; left-multiply the error by a diagonal matrix of activation magnitudes before SVD so its singular values decay faster, needing only 32 calibration samples and no training.
Key results
- L²QER-MXINT W4A8 reaches WikiText-2 perplexity about 0.15 above FP16 on average and about 0.3% average downstream accuracy drop on LLaMA models.
- On OPT-1.3B W4A8, plain MXINT gives PPL 16.42, LQER 15.28, and L²QER 15.02 versus FP16 14.63.
- L²QER-INT W4A8 with 0.33x circuit area of FP16-relative baselines versus 13.99x for AWQ-style dequantization circuit area (reported as relative area).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add low-rank quantization-error reconstruction (ablation) | Plain MXINT W4A8 on OPT-1.3B · WikiText-2 | perplexity: 16.42 -> 15.28 | Helped |
| Add the activation-induced scale matrix S to LQER (ablation) | LQER W4A8 on OPT-1.3B · WikiText-2 | perplexity: 15.28 -> 15.02 | Helped |
| Add low-rank quantization-error reconstruction (ablation) | Plain MXINT W4A8 on LLaMA-7B · WikiText-2 | perplexity: 6.17 -> 6.06 | Helped |
| Add the activation-induced scale matrix S to LQER (ablation) | LQER W4A8 on LLaMA-7B · WikiText-2 | perplexity: 6.06 -> 5.89 | Helped |
| Use L²QER-MXINT W4A8 instead of AWQ INT4 g128 | LLaMA-7B · WikiText-2 | perplexity: 5.20 -> 5.21 | No clear effect |
| Use L²QER-MXINT W4A8 instead of AWQ INT4 g128 | LLaMA-7B · Six downstream tasks | average accuracy: 62.5% -> 63.0% | Helped |
| Reduce activation precision from 8 bits to 6 bits (ablation) | L²QER-MXINT on LLaMA-7B · WikiText-2 | perplexity: 5.21 -> 5.24 | Hurt |
| Quantize weights to 2 bits using L²QER with rank k=256 | LLaMA-7B · WikiText-2 | perplexity: 5.67 -> 10.30 | Hurt |
Surprising findings
- OmniQuant reproduced by the authors degraded substantially on LLaMA downstream tasks (about 11% average drop), contrary to its reported results.
- 2-bit quantization remains challenging: L²QER at W2A8 with rank 256 gives perplexity 10.30 on LLaMA-7B, with AWQ 2-bit collapsing to 2.6e5.
Limitations
- The scaling matrix S is an empirical heuristic without analytical derivation.
- 2-bit quantization remains challenging and needs larger rank (k=256).
- Hardware cost comparisons are FPGA LUT-based estimates, and many baseline numbers are taken from original papers rather than rerun.
Experimental setup
Datasets: SlimPajama (Wikipedia excluded, calibration subset) WikiText-2 ARC (easy) ARC (challenge) LAMBADA PIQA OpenBookQA BoolQ AlpacaEval
Benchmarks: WikiText-2 ARC (easy) ARC (challenge) LAMBADA PIQA OpenBookQA BoolQ AlpacaEval
Metrics: perplexity accuracy length-controlled win rate win rate average weight bitwidth FPGA circuit area (LUTs)
Baselines: GPTQ AWQ LLM.int4() OmniQuant AQAS QuIP# SmoothQuant
Builds on: LLM.int8() SmoothQuant AWQ LoftQ MXINT
Closest papers by meaning
- ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization (AAAI 2025 · similarity 0.93)
- QERA: an Analytical Framework for Quantization Error Reconstruction (ICLR 2025 · similarity 0.92)
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.91)
- RILQ: Rank-Insensitive LoRA-Based Quantization Error Compensation for Boosting 2-Bit Large Language Model Accuracy (AAAI 2025 · similarity 0.91)
- LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning (ICLR 2024 · similarity 0.91)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.