Research map / Architectures and efficiency / Quantization
QuIP$\#$: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces QuIP#, a weight-only post-training quantization method combining randomized Hadamard incoherence processing, E8-lattice-based vector codebooks (E8P) with BlockLDLQ rounding, and inter-layer fine-tuning, reaching state-of-the-art Llama 1/2 perplexity at 2–4 bits.
Problem
Post-training weight quantization of LLMs to extreme low-bit regimes (≤4 bits per weight) while keeping quality high and decoding fast on GPUs.
Approach
Replace QuIP's Kronecker incoherence transform with a randomized Hadamard transform, round weights with block adaptive rounding (BlockLDLQ) using a structured E8 lattice codebook (E8P) that fits in L1 cache, and fine-tune the unquantized parameters layer-wise and end-to-end during quantization.
Key results
- At 2 bits on Llama 2 70B, QuIP# reaches 4.16 WikiText2 perplexity at context 2048 (vs. OmniQuant 2-bit unusable), and 3-bit QuIP# outperforms theoretical lossless 4-bit FP16 scaling on Llama 2.
- QuIP# 2-bit on RTX 4090 reaches 170.5 tok/s on 2-7B and over 50% of peak memory bandwidth on 2-70B (56.84%) with a FlashAttention-based implementation.
- On HuggingFace Llama inference, QuIP# 2-bit achieves 106.3 tok/s on 2-7B vs. 33.1 tok/s for FP16 and 20.6 tok/s for AQLM 2-bit.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QuIP# 2-bit instead of AQLM 2.07-bit | Llama 2 70B · WikiText2 (ctx 4096) | perplexity: 3.94 -> 3.91 | Helped |
| Use QuIP# 4-bit instead of AQLM 4.07-bit | Llama 2 70B · WikiText2 (ctx 4096) | perplexity: 3.17 -> 3.18 | Hurt |
| Add fine-tuning (ablation) | QuIP# 2-bit on Llama 2 7B · WikiText2 (ctx 4096) | perplexity: 8.22 -> 6.19 | Helped |
| Add E8 lattice codebooks and fine-tuning (ablation) | QuIP# 2-bit on Llama 2 7B · WikiText2 (ctx 4096) | perplexity: 11.2 -> 6.19 | Helped |
| Add fine-tuning (ablation) | QuIP# 2-bit on Llama 2 13B · WikiText2 (ctx 4096) | perplexity: 6.06 -> 5.35 | Helped |
| Add E8 lattice codebooks and fine-tuning (ablation) | QuIP# 2-bit on Llama 2 13B · WikiText2 (ctx 4096) | perplexity: 7.04 -> 5.35 | Helped |
| Add fine-tuning (ablation) | QuIP# 2-bit on Llama 2 70B · WikiText2 (ctx 4096) | perplexity: 4.16 -> 3.91 | Helped |
| Add E8 lattice codebooks and fine-tuning (ablation) | QuIP# 2-bit on Llama 2 70B · WikiText2 (ctx 4096) | perplexity: 4.58 -> 3.91 | Helped |
Surprising findings
- 3-bit QuIP# on Llama 2 outperforms a theoretical lossless 4-bit FP16 model in perplexity scaling.
- E8P codebook outperforms a K-means codebook of the same dimension and bitrate, despite K-means being lower MSE on the source distribution.
Limitations
- Fine-tuning and RVQ-based 3/4-bit results are not the most advanced multi-codebook approaches; authors note better numbers may be possible.
- Some design choices (fast inference) restrict achievable quantization quality; the CUDA kernel is a proof of concept.
- Hadamard dimensions not power of two require factorization with known Hadamard matrices; RFFT alternative performs slightly worse.
Experimental setup
Datasets: WikiText2 C4
Benchmarks: WikiText2 C4 ArcC ArcE PiQA Winogrande BoolQ
Metrics: perplexity zero-shot accuracy tokens per second percent of peak memory bandwidth
Baselines: AWQ OmniQuant QuIP AQLM SpQR SqueezeLLM LLM-QAT
Builds on: QuIP LDLQ AQLM OPTQ
Closest papers by meaning
- Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment (NeurIPS 2025 · similarity 0.90)
- Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression (NeurIPS 2025 · similarity 0.90)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.89)
- PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression (NeurIPS 2024 · similarity 0.89)
- ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization (NeurIPS 2025 · similarity 0.88)
- LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.