Research map / Architectures and efficiency / Quantization
GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GPTAQ, a finetuning-free layer-wise weight quantization method that calibrates each layer to match full-precision layer outputs (asymmetric calibration), extending GPTQ with a closed-form correction term and efficient GPU parallelization.
Problem
GPTQ-style layer-wise calibration matches the quantized layer's output using the quantized model's own inputs, so quantization error accumulates across layers; finetuning-based fixes are too costly for very large transformers.
Approach
Calibrate each layer against the full-precision model's output using full-precision inputs (asymmetric calibration). The optimal update adds a residual-compensation term to GPTQ's update, made tractable by neuron decomposition of the residual, Cholesky reformulation, and a masked matrix product P computed once per layer.
Key results
- On W4A4 QuaRot LLaMA3-70B, GPTAQ reaches 6.93 Wikitext2 perplexity versus 9.44 for QuaRot+GPTQ.
- On LLaMA3.1-405B with W4A4 on a single A100, GPTAQ achieves 3.48 perplexity versus 5.82 for GPTQ (RTN 17.4).
- On 3-bit per-group weight-only LLaMA3-8B-Instruct, GPTAQ reaches 63.8% average accuracy, 2.4% above AWQ and 1.3% above GPTQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use asymmetric calibration instead of GPTQ | DeiT-S, W4A4 · DeiT-S | ImageNet accuracy: 71.9 -> 72.8 | Helped |
| Use asymmetric calibration instead of GPTQ | DeiT-B, W4A4 · DeiT-B | ImageNet accuracy: 77.7 -> 78.4 | Helped |
| Use asymmetric calibration with QuaRot instead of GPTQ | LLaMA3-8B, W4A4 · LLaMA3-8B | Wikitext2 perplexity: 7.80 -> 7.37 | Helped |
| Use asymmetric calibration with QuaRot instead of GPTQ | LLaMA3-70B, W4A4 · LLaMA3-70B | Wikitext2 perplexity: 9.44 -> 6.93 | Helped |
| Use asymmetric calibration with QuaRot instead of GPTQ | LLaMA2-7B, W4A4 · LLaMA2-7B | Wikitext2 perplexity: 6.00 -> 5.85 | Helped |
| Use asymmetric calibration instead of GPTQ | EVA-02, W4A4 · EVA-02 | ImageNet top-1 accuracy: 86.48 -> 88.30 | Helped |
| Use asymmetric calibration instead of GPTQ | LLaMA3.1-405B, W4A4 · LLaMA3.1-405B | Wikitext2 perplexity: 5.82 -> 3.48 | Helped |
| Add the residual-error update term to the quantization-error update (ablation) | GPTAQ on LLaMA3-8B · LLaMA3-8B, W4A4 | Wikitext2 perplexity: 7.80 -> 7.36 | Helped |
| Add the residual-error update term to the quantization-error update (ablation) | GPTAQ on LLaMA3-8B · LLaMA3-8B, W4A4 | zero-shot average accuracy: 67.1 -> 69.6 | Helped |
| Use only the residual-error update term instead of both update terms (ablation) | GPTAQ on LLaMA3-8B · LLaMA3-8B, W4A4 | Wikitext2 perplexity: 7.36 -> 7.97 | Hurt |
| Use only the residual-error update term instead of both update terms (ablation) | GPTAQ on LLaMA3-8B · LLaMA3-8B, W4A4 | zero-shot average accuracy: 69.6 -> 69.0 | Hurt |
| Use only the quantization-error update term instead of both update terms (ablation) | GPTAQ on LLaMA3-8B · LLaMA3-8B, W4A4 | Wikitext2 perplexity: 7.36 -> 7.80 | Hurt |
Surprising findings
- The second (residual-compensation) term applied alone outperforms GPTQ's first term in average accuracy on LLaMA3-8B W4A4 (69.0 vs 67.1), despite worse perplexity.
Limitations
- Extra memory for storing full-precision inputs and the P matrix during calibration.
- Results on some settings (e.g., SpinQuant+GPTAQ latency) add 30-40% overhead for larger layers.
- Gains are reported mainly on perplexity and zero-shot tasks with 128 calibration samples; generality beyond these settings is not extensively explored.
Experimental setup
Datasets: WikiText2 C4 ImageNet
Benchmarks: WikiText2 perplexity C4 perplexity PiQA ARC-Easy ARC-Challenge HellaSwag Winogrande BoolQ OBQA SiQA ImageNet top-1
Metrics: perplexity accuracy average accuracy GPU hours
Baselines: GPTQ RTN AWQ OmniQuant QLLM DuQuant QuaRot SpinQuant PTQ4ViT APQ-ViT PD-Quant RepQ-ViT
Builds on: GPTQ Optimal Brain Compression QuaRot SpinQuant
Closest papers by meaning
- GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers (NeurIPS 2025 · similarity 0.89)
- ERQ: Error Reduction for Post-Training Quantization of Vision Transformers (ICML 2024 · similarity 0.87)
- FrameQuant: Flexible Low-Bit Quantization for Transformers (ICML 2024 · similarity 0.87)
- FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation (CVPR 2025 · similarity 0.87)
- Outlier-aware Slicing for Post-Training Quantization in Vision Transformer (ICML 2024 · similarity 0.87)
- The Quantization Benefits of Residual-Free Transformers (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.