Research map / Architectures and efficiency / Quantization
GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GANQ, a layer-wise post-training non-uniform (LUT-based) weight quantization method that solves a decomposable mixed-integer quadratic problem with GPU-parallel alternating optimization, reporting lower perplexity than RTN/GPTQ/OmniQuant and up to 2.57x speedup on an RTX 4090.
Problem
Weight-only LLM quantization suffers from mixed-precision GEMM lacking hardware support and from uniform quantization grids that poorly fit non-uniform LLM weight distributions, causing accuracy loss at 3-4 bits.
Approach
Formulate per-row LUT-based non-uniform quantization as a layer-wise output-error minimization (mixed-integer quadratic program), decompose it into independent row subproblems, and solve it with alternating updates: a Cholesky/back-substitution assignment of codebook indices and a closed-form least-squares codebook update, batched on GPU.
Key results
- On WikiText-2 with 4-bit quantization, GANQ reaches 6.61 on LLaMA-3-8B vs 6.13 FP16 and 8.53 RTN; for LLaMA-2-7B 4-bit average zero-shot accuracy is 64.23% vs 64.47% FP16.
- On LLaMA-2-7B 3-bit zero-shot mean accuracy is 62.22%, above OmniQuant (58.66%) and GPTQ (54.61%).
- On RTX 4090, GANQ 3-bit achieves up to 2.57x speedup over FP16 on OPT-6.7B (CUDA time 6.51s vs 16.76s) with peak memory 4.10 GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GANQ instead of OmniQuant for 4-bit weight quantization | OPT-6.7B · WikiText-2 | perplexity: 11.25 -> 10.70 | Helped |
| use GANQ instead of OmniQuant for 3-bit weight quantization | OPT-6.7B · WikiText-2 | perplexity: 13.47 -> 11.39 | Helped |
| use GANQ instead of OmniQuant for 4-bit quantization | LLaMA-2-7B · zero-shot tasks | mean accuracy: 62.94 -> 64.23 | Helped |
| use GANQ instead of OmniQuant for 3-bit quantization | LLaMA-2-7B · zero-shot tasks | mean accuracy: 58.66 -> 62.22 | Helped |
| use GANQ instead of OmniQuant for 4-bit quantization | LLaMA-3.2-1B-Instruct · GSM8K | accuracy: 16.53 -> 27.75 | Helped |
| use GANQ instead of FP16 inference | OPT-6.7B · generation, 3-bit | speedup: 1.0 -> 2.57 | Helped |
| integrate outlier handling with GANQ | LLaMA-7B and LLaMA-2-7B · WikiText-2 perplexity | perplexity: null | Mixed |
| replace adaptive preconditioning with fixed lambda preconditioning (ablation) | GANQ on OPT-125M · WikiText-2, 4-bit | perplexity: 28.58 -> 29.14; 28.58 -> 29.04; 28.58 -> 28.98; 28.58 -> 29.05; 28.58 -> 29.09 | Hurt |
Surprising findings
- On OPT-2.7B, 4-bit GANQ perplexity (12.33) is slightly lower than the FP16 model (12.47).
- GPTQ 4-bit on LLaMA-3-8B yields WikiText-2 perplexity of 2.4e2, indicating severe collapse of that baseline.
Limitations
- Relies on LUT-based inference kernels for speedups; GANQ* outlier variant adds latency from sparse operations.
- Some baseline results (e.g., SqueezeLLM) are taken from original papers, and SqueezeLLM is limited to models up to 2.7B in the authors' runs.
- 3-bit GANQ on LLaMA-3.2 models still shows large perplexity gaps versus FP16 (e.g., 15.91 vs 9.76 on 1B WikiText-2).
Experimental setup
Datasets: C4 WikiText-2 PTB
Benchmarks: WikiText-2 C4 PTB ARC Easy ARC Challenge WinoGrande BoolQ RTE HellaSwag GSM8K LongBench
Metrics: perplexity accuracy CUDA time speedup peak memory
Baselines: RTN GPTQ OmniQuant AWQ SqueezeLLM
Builds on: GPTQ SqueezeLLM OmniQuant AWQ
Closest papers by meaning
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.91)
- SqueezeLLM: Dense-and-Sparse Quantization (ICML 2024 · similarity 0.90)
- LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.89)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.89)
- LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning (ICLR 2024 · similarity 0.89)
- LoQT: Low-Rank Adapters for Quantized Pretraining (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.