Research map / Architectures and efficiency / Quantization
LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LUT-GEMM, a lookup-table kernel that runs matrix-vector products directly on binary-coding-quantized (BCQ) weights with full-precision activations, removing dequantization and enabling single-GPU inference of OPT-175B.
Problem
Weight-only quantized LLM inference in the memory-bound generation phase relies on dequantization kernels (OPTQ, AWQ) that add overhead, and multi-GPU model parallelism is needed for large models in FP16.
Approach
Extend binary-coding quantization with a bias term so it represents both uniform and non-uniform quantization, then compute BCQ matrix-vector products via precomputed lookup tables of activation sub-vector combinations, avoiding dequantization; add group-wise scaling factors for a finer compression/accuracy trade-off.
Key results
- On OPT-175B with 3-bit quantization, LUT-GEMM runs on a single A100 80GB GPU at 51.6 ms/token versus 106.5 ms for OPTQ on one GPU, a 2.1x latency reduction.
- Kernel-level, the first FFN layer of OPT-175B runs in 0.2250 ms with INT3 LUT-GEMM versus 0.7256 ms for cuBLAS FP16 (3.22x speed-up).
- OPT-175B FP16 requires 8 GPUs (42.4 ms/token), while 3-bit LUT-GEMM fits on one GPU with 51.6 ms/token, and 2-bit group-wise settings reach 6.40-7.53x compression.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace dequantization-plus-GEMM with direct LUT-GEMM on quantized weights | OPT-175B FFN first layer · OPT-175B FFN first layer, INT3 weights | latency: 0.3599 -> 0.2250 ms | Helped |
| use LUT-GEMM instead of cuBLAS tensor parallelism across 8 GPUs | OPT-175B matrix multiplication · OPT-175B, q=2, 1 GPU vs cuBLAS q=16, 8 GPUs | total energy: 299.96 -> 43.49 mJ | Helped |
| quantize weights from 4-bit to 3-bit (ablation) | LUT-GEMM AWQ · LLaMA-30B, 1 GPU | latency: 20.7 -> 18.1 ms | Helped |
| quantize weights from 4-bit to 3-bit (ablation) | LUT-GEMM AWQ · LLaMA-30B, Wiki2 | perplexity: 4.23 -> 4.88 | Hurt |
| quantize weights from 4-bit to 3-bit (ablation) | LUT-GEMM AWQ · LLaMA-65B, 2 GPUs | latency: 25.4 -> 23.0 ms | Helped |
| quantize weights from 4-bit to 3-bit (ablation) | LUT-GEMM AWQ · LLaMA-65B, Wiki2 | perplexity: 3.67 -> 4.24 | Hurt |
| reduce quantization from OPTQ 3-bit to LUT-GEMM 1-bit (ablation) | OPT-175B generation · OPT-175B, 1 GPU | latency per token: 106.5 -> 30.4 ms | Helped |
| reduce quantization from OPTQ 3-bit to LUT-GEMM 1-bit (ablation) | OPT-175B generation · OPT-175B, 1 GPU | perplexity | Mixed |
| reduce group size from g=128 to g=32 at 2-bit quantization (ablation) | LUT-GEMM OPTQ · OPT-175B, 1 GPU | latency: 46.5 -> 55.2 ms | Hurt |
| reduce group size from g=128 to g=32 at 2-bit quantization (ablation) | LUT-GEMM OPTQ · OPT-175B, Wiki2 | perplexity: 9.58 -> 8.94 | Helped |
| use smaller group size g (ablation) | RTN-quantized OPT models · OPT-30B and OPT-66B | perplexity | Mixed |
Surprising findings
- Multi-GPU tensor parallelism with cuBLAS gives far smaller speed-ups than GPU count, and LUT-GEMM makes communication overhead more pronounced, reducing the benefit of model parallelism.
Limitations
- Gains diminish as batch size increases, due to shared-memory bandwidth limits between cores and LUTs.
- Evaluated mainly on single-batch inference on A100 GPUs.
- Group size and bit-width require hyperparameter search to balance compression and accuracy.
Experimental setup
Datasets: WikiText-2 PTB C4 LAMBADA PIQA MNLI
Benchmarks: WikiText-2 PTB C4 LAMBADA PIQA MNLI
Metrics: latency per token (ms) perplexity accuracy compression ratio speed-up
Baselines: OPTQ AWQ cuBLAS FP16 cuBLAS INT8
Builds on: binary-coding quantization OPTQ AWQ FasterTransformer
Closest papers by meaning
- SqueezeLLM: Dense-and-Sparse Quantization (ICML 2024 · similarity 0.89)
- GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models (ICML 2025 · similarity 0.88)
- KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.88)
- CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs (NeurIPS 2025 · similarity 0.88)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.88)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.