Research map / Architectures and efficiency / Quantization
ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ABQ-LLM, a PTQ method with distribution correction, bit balance for INT2, and a GPU kernel that decomposes arbitrary-bit quantized matmul into 1-bit binary tensor core operations.
Problem
Low-bit weight-activation PTQ of LLMs degrades performance, and GPU integer tensor cores (INT4/INT8 only) cannot efficiently accelerate arbitrary precision combinations or GEMV-shaped decoding workloads.
Approach
Combine a block-wise distribution correction (DLC loss on down_proj and block outputs plus an attention-aware KL loss) and a symmetric-range fix for INT2 (bit balance) with a kernel that decomposes any p-bit by q-bit matmul into p×q binary tensor core (BMMA) operations plus shifts, removing padding waste in GEMV.
Key results
- On LLaMA-7B W2*A8, ABQ-LLM reaches WikiText2 perplexity 7.59 versus 9.76 for AffineQuant (W2A8 variant reported at 11.35 for ABQ-LLM without bit balance).
- Versus SmoothQuant in FasterTransformer end-to-end inference, ABQ-LLM reports 1.6x speedup and 2.7x memory compression.
- ABQKernel reports 7.47x throughput gain over W8A8 CUTLASS/cuBLAS on GEMV (1,4096)x(4096,4096) for W2A8 on RTX 3070/4080 per the paper's kernel benchmark.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply bit balance strategy to W2A8 (ablation) | ABQ-LLM on LLaMA-7B · WikiText2 | perplexity: 11.35 -> 7.59 | Helped |
| Apply bit balance strategy to W2A16 (ablation) | ABQ-LLM on LLaMA-7B · WikiText2 | perplexity: 11.48 -> 7.50 | Helped |
| Apply bit balance strategy (ablation) | ABQ-LLM on LLaMA-13B · zero-shot tasks | average accuracy: 56.15 -> 61.96 | Helped |
| Add pipeline optimization (ablation) | ABQKernel · (1, 4096) × (4096, 4096) GEMV | latency (us): 20.05 -> 14.66 | Helped |
| Add GEMV elimination (ablation) | ABQKernel · (1, 4096) × (4096, 4096) GEMV | latency (us): 14.66 -> 10.92 | Helped |
| Add auto kernel search (ablation) | ABQKernel · (1, 4096) × (4096, 4096) GEMV | latency (us): 10.92 -> 6.68 | Helped |
| Use ABQ-LLM instead of AffineQuant | LLaMA-7B W2A8 · WikiText2 | perplexity: 9.76 -> 11.35 | Hurt |
Surprising findings
- Bit balance on INT2 weights substantially improves perplexity (e.g., LLaMA-7B W2A16 11.48 to W2*A16 7.50), beating AffineQuant's 9.53 only partially and matching W3A16 levels.
- AffineQuant's W2A8 result on LLaMA-2-7B is extremely unstable (perplexity 1483 on WikiText2), showing baseline collapse at very low bits.
Limitations
- Hardware evaluations are on specific GPUs (RTX 3070/4080, A800) and FasterTransformer integration; results may not transfer to other stacks.
- Baseline methods could not be fully reproduced under the authors' compute limits, so some comparisons rely on reported numbers.
- Calibration relies on per-token and per-channel settings; per-group results are only shown for W4A4 g128 on LLaMA-7B.
Experimental setup
Datasets: WikiText2
Benchmarks: WikiText2 C4 PIQA ARC-e ARC-c BoolQ HellaSwag Winogrande
Metrics: perplexity zero-shot accuracy GEMV latency TOPS throughput memory usage
Baselines: SmoothQuant OmniQuant I-LLM AffineQuant GPTQ AWQ Atom CUTLASS cuBLAS
Builds on: OmniQuant AffineQuant SmoothQuant GPTQ AWQ
Closest papers by meaning
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.93)
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.92)
- SqueezeLLM: Dense-and-Sparse Quantization (ICML 2024 · similarity 0.92)
- SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models (ICML 2025 · similarity 0.92)
- CBQ: Cross-Block Quantization for Large Language Models (ICLR 2025 · similarity 0.92)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.