Research map / Architectures and efficiency / Quantization
GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GuidedQuant, a post-training quantization objective that weights layer output errors by end-loss gradients while keeping block-diagonal Fisher interactions within output channels, plus LNQ, a non-uniform scalar quantizer, and shows gains across weight-only scalar, vector, and W4A4 quantization of Llama-2 models.
Problem
Post-training quantization of LLMs usually minimizes layer-wise output error that treats all hidden features equally or uses diagonal-Fisher surrogates, ignoring both end-loss sensitivity and cross-weight interactions.
Approach
Scale layer output errors by end-loss gradients and approximate the Fisher with block-diagonal per-output-channel blocks, then average those blocks within a small number of channel groups so they are tractable at LLM scale; applicable as a plug-in to any layer-wise output-based PTQ method.
Key results
- LNQ + GuidedQuant at 2.01 bits on Llama-2-7B reaches 8.83 Wiki2 perplexity versus 39.58 for SqueezeLLM.
- QTIP + GuidedQuant at 2 bits on Llama-2-7B improves Wiki2 perplexity from 6.82 to 6.11.
- SpinQuant + GuidedQuant at W4A4KV4 on Llama-2-7B improves Wiki2 (2048 context) perplexity from 5.95 to 5.89.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GuidedQuant to LNQ | LNQ weight-only scalar quantization · WikiText2, Llama-2-7B, 2-bit | perplexity: 23.31 -> 8.83 | Helped |
| Add GuidedQuant to LNQ | LNQ weight-only scalar quantization · WikiText2, Llama-2-13B, 2-bit | perplexity: 8.78 -> 7.26 | Helped |
| Add GuidedQuant to LNQ | LNQ weight-only scalar quantization · WikiText2, Llama-2-70B, 2-bit | perplexity: 5.23 -> 5.04 | Helped |
| Add GuidedQuant to QTIP | QTIP weight-only vector quantization · WikiText2, Llama-2-7B, 2-bit | perplexity: 6.82 -> 6.11 | Helped |
| Add GuidedQuant to QTIP | QTIP weight-only vector quantization · WikiText2, Llama-2-13B, 2-bit | perplexity: 5.52 -> 5.33 | Helped |
| Add GuidedQuant to QTIP | QTIP weight-only vector quantization · WikiText2, Llama-2-70B, 2-bit | perplexity: 3.87 -> 3.80 | Helped |
| Add GuidedQuant to SpinQuant | SpinQuant weight-and-activation quantization · WikiText2, Llama-2-7B, W4A4KV4 | perplexity: 5.95 -> 5.89 | Helped |
| Add GuidedQuant to SpinQuant | SpinQuant weight-and-activation quantization · WikiText2, Llama-2-13B, W4A4KV4 | perplexity: 5.24 -> 5.19 | Helped |
| Add GuidedQuant to SpinQuant | SpinQuant weight-and-activation quantization · WikiText2, Llama-2-70B, W4A4KV4 | perplexity: 3.71 -> 3.71 | No clear effect |
| Replace GPTVQ 1D optimization with LNQ (ablation) | Weight-only scalar quantization · WikiText2, Llama-2-13B, 2-bit | perplexity: 9.53 -> 8.78 | Helped |
| Use cyclic coordinate descent instead of GPTQ for assignments (ablation) | LNQ non-uniform scalar quantization · Non-uniform scalar quantization; reported ablation setting | perplexity | Mixed |
Limitations
- Ignores cross-layer and cross-output-channel interactions in the Fisher approximation.
- Requires a backward pass on calibration data and grouping hyperparameter g; quantization cost grows with g.
- Gains reported only on Llama-2 models and calibration-based settings; fairness impacts of quantization not evaluated.
Experimental setup
Datasets: RedPajama WikiText2
Benchmarks: WikiText2 C4
Metrics: perplexity tokens per second
Baselines: SqueezeLLM GPTVQ 1D GPTQ QuIP QuIP# AQLM QTIP QuaRot SpinQuant GPTVQ 2D GPTVQ 4D
Builds on: SqueezeLLM GPTQ QTIP SpinQuant GPTVQ
Closest papers by meaning
- LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.91)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.90)
- OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting (ICLR 2025 · similarity 0.90)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.90)
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.90)
- Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.