Research map / Architectures and efficiency / Quantization
IM-Unpack: Training and Inference with Arbitrarily Low Precision Integers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that round-to-nearest integer GEMMs reach near-full-precision parity for Transformer inference and training, and proposes IM-Unpack, which unpacks out-of-range "heavy hitter" integers into larger matrices so that exact GEMMs can run purely on arbitrarily low bit-width integers.
Problem
Low bit-width integer GEMMs are efficient but suffer large rounding errors because a few heavy-hitter entries in Transformer matrices far exceed the representable range of low bit-width integers, hurting both inference and training.
Approach
Decompose out-of-bound integer entries into multiple in-bound low bit-width integers by row/column unpacking (chosen per matrix via outlier counts), so the original integer GEMM is reproduced exactly using only low bit-width integer GEMMs followed by bit-shift scaling and accumulation.
Key results
- RTN with beta=31 on LLaMA-7B matches BF16 zero-shot accuracy closely (e.g., WinoGrande 69.3 vs 68.8 BF16) when quantizing linear layers.
- RoBERTa-Small RTN quantized training with beta=31 reaches validation log perplexity 1.840 vs 1.869 for FP32.
- Unpack ratios for Mix strategy on LLaMA-7B linear layers are about 1.27 at 4 bits with beta=5-ish settings, with attention-output GEMMs showing higher overhead (~3-4x at 5 bits).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RTN with β=15 for linear-layer computations instead of full precision | LLaMA-7B · Zero-shot ARC-c | accuracy: 43.1 -> 43.0 | No clear effect |
| Use RTN with β=15 for all GEMMs instead of full precision | LLaMA-7B · Zero-shot ARC-c | accuracy: 43.1 -> 44.0 | Helped |
| Train with RTN quantized GEMMs using β=31† instead of FP32 | ViT-Small · ImageNet validation | top-1 accuracy: 78.91 -> 79.33 | Helped |
| Train with RTN quantized GEMMs using β=31 instead of FP32 | RoBERTa-Small · Validation | log perplexity: 1.869 -> 1.840 | Helped |
| Fine-tune with RTN quantized GEMMs using β=31 instead of BF16 | T5-Large · XSum validation | Rouge2: 13.00 -> 13.83 | Helped |
| Increase β from 5 to 15 for linear-layer RTN (ablation) | LLaMA-7B · Zero-shot ARC-c | accuracy: 39.3 -> 43.0 | Helped |
| Restrict all values to the β=255 representable range (p=100%) instead of quantizing at p=95%, β=31 (ablation) | LLaMA-7B · Zero-shot ARC-c | accuracy: 43.4 -> 35.8 | Hurt |
| Clip outliers at the 99.5-percentile instead of quantizing at p=95%, β=31 (ablation) | LLaMA-7B · Zero-shot ARC-c | accuracy: 43.4 -> 21.4 | Hurt |
| Use row unpacking for both matrices rather than Mix (ablation) | IM-Unpack on LLaMA-7B · Linear-layer GEMMs, β=15 and b=5 | unpack ratio: 1.06 -> 2.02 | Hurt |
| Use row unpacking for A and column unpacking for B rather than Mix (ablation) | IM-Unpack on LLaMA-7B · Linear-layer GEMMs, β=15 and b=5 | unpack ratio: 1.06 -> 5.36 | Hurt |
| Use column unpacking for A and row unpacking for B rather than Mix (ablation) | IM-Unpack on LLaMA-7B · Linear-layer GEMMs, β=15 and b=5 | unpack ratio: 1.06 -> 1.64 | Hurt |
| Use column unpacking for both matrices rather than Mix (ablation) | IM-Unpack on LLaMA-7B · Linear-layer GEMMs, β=15 and b=5 | unpack ratio: 1.06 -> 4.01 | Hurt |
Surprising findings
- Simply enlarging the integer bit-width to cover heavy hitters (clipping-free, beta=255) causes a large performance drop, so widening bits alone does not fix the problem.
- Clipping outliers at the 99.5th percentile causes catastrophic degradation (e.g., LLaMA-7B ARC-e 25.5).
Limitations
- Unpacking is inefficient for the self-attention matrix M whose outliers concentrate on the diagonal rather than rows or columns.
- Only simple RTN quantization is used, so results may not be optimal; large beta is still needed for some ViT gradients.
- Overhead of unpacking grows with the number of heavy hitters, so efficiency gains are not guaranteed for all matrices.
Experimental setup
Datasets: English Wikipedia ImageNet XSum ARC-c ARC-e BoolQ HellaSwag PIQA WinoGrande
Benchmarks: ARC-c ARC-e BoolQ HellaSwag PIQA WinoGrande ImageNet
Metrics: zero-shot accuracy top-1 accuracy validation log perplexity unpack ratio ROUGE loss
Baselines: LLM.int8() SmoothQuant LLM-QAT LLM-FP4 FQ-ViT I-ViT PTQ4ViT APQ-ViT RepQ-ViT GPTQ QuIP
Builds on: LLM.int8() SmoothQuant RoBERTa ViT
Closest papers by meaning
- LibAMM: Empirical Insights into Approximate Computing for Accelerating Matrix Multiplication (NeurIPS 2024 · similarity 0.86)
- CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs (NeurIPS 2025 · similarity 0.85)
- An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networks (ICML 2025 · similarity 0.84)
- QiMeng-GEMM: Automatically Generating High-Performance Matrix Multiplication Code by Exploiting Large Language Models (AAAI 2025 · similarity 0.84)
- A Fast Convoluted Story: Scaling Probabilistic Inference for Integer Arithmetics (NeurIPS 2024 · similarity 0.84)
- QBB: Quantization with Binary Bases for LLMs (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.