Research map / Architectures and efficiency / Quantization
NestQuant: nested lattice quantization for matrix products and LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NestQuant, a post-training quantizer for weights, activations and KV cache built on nested Gosset (E8) lattice codes with multi-beta shaping and QA-LDLQ, reaching ~6.6 wikitext2 perplexity on 4-bit Llama-3-8B.
Problem
Full 4-bit post-training quantization of LLMs (weights, KV cache and activations) loses significant accuracy with uniform scalar quantizers; matrix-product quantization is far from information-theoretic limits.
Approach
Apply nested-lattice Voronoi codes built on the 8-dimensional Gosset lattice, with a union of several scaling coefficients (betas) to limit overload, a fast Gosset oracle for runtime decoding, and QA-LDLQ, a weight-rounding correction that accounts for quantized activations.
Key results
- NestQuant W4A4KV4 on Llama-3-8B reaches 6.63 wikitext2 perplexity versus 7.3 for SpinQuant and OstQuant and 8.16 for QuaRot (FP 6.14).
- On Llama-3-70B at W4A4KV4, NestQuant reaches 3.61 perplexity versus 6.66 for QuaRot (FP 2.86).
- Weight-only 4-bit (q=14, k=4) on Llama-3-8B reaches 6.31 perplexity at 3.99 bits per entry, versus 6.5 for SpinQuant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NestQuant instead of SpinQuant for 4-bit weights, KV cache, and activations | Llama-3-8B · Wikitext2 | perplexity: 7.3 -> 6.6 | Helped |
| Use NestQuant instead of SpinQuant for 4-bit weights only | Llama-3-8B · Wikitext2 | perplexity: 6.5 -> 6.3 | Helped |
| Use NestQuant instead of SpinQuant for 4-bit weights and KV cache | Llama-3-8B · Wikitext2 | perplexity: 6.6 -> 6.4 | Helped |
| Use NestQuant instead of SpinQuant for 4-bit full quantization | Llama-3-8B · Winogrande | accuracy: 0.66 -> 0.72 | Helped |
| Use NestQuant instead of SpinQuant for 4-bit weights only | Llama-3-8B · ARC-Challenge | accuracy: 0.54 -> 0.53 | Hurt |
| Add LDLQ (ablation) | NestQuant q=14,k=4 on Llama-3-8B · Wikitext2, weights + KV cache + activations | perplexity: 6.849 -> 6.633 | Helped |
| Replace the standard decoder with NestQuantM's simplified decoder (ablation) | NestQuant on Llama-3-8B · Wikitext2, weights + KV cache + activations | perplexity: 6.63 -> 6.82 | Hurt |
| Reduce q from 14 to 12 (ablation) | NestQuant on Llama-3-8B · Wikitext2, weights + KV cache + activations | perplexity: 6.633 -> 6.841 | Hurt |
| Reduce q from 14 to 10 (ablation) | NestQuant on Llama-3-8B · Wikitext2, weights + KV cache + activations | perplexity: 6.633 -> 7.251 | Hurt |
| Reduce q from 14 to 8 (ablation) | NestQuant on Llama-3-8B · Wikitext2, weights + KV cache + activations | perplexity: 6.633 -> 7.989 | Hurt |
Surprising findings
- Original LDLQ on Llama-3-70B yields perplexity around 10^4 at W4A4 due to outliers in one layer, which QA-LDLQ fixes.
- Without LDLQ, NestQuant still reaches 6.85 at W+KV+A, which is still better than prior methods' ~7.3.
Limitations
- Uses a Gosset lattice with d=8 rather than a higher-dimensional lattice, so the granular gain remains below the information-theoretic bound.
- Fine-tuning-based refinements such as rotation optimization were deliberately omitted, so results are not SOTA-tuned.
- Multi-beta union adds rate overhead and requires dynamic programming calibration per layer.
Experimental setup
Datasets: WikiText2
Benchmarks: WikiText2 perplexity ARC-Easy ARC-Challenge HellaSwag PIQA Winogrande
Metrics: perplexity zero-shot accuracy bits per entry RMSE
Baselines: SpinQuant QuaRot OstQuant QuIP# DuQuant GPTQ LLM-QAT
Builds on: Voronoi codes LDLQ QuaRot SpinQuant QuIP#
Closest papers by meaning
- SpinQuant: LLM Quantization with Learned Rotations (ICLR 2025 · similarity 0.90)
- Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression (NeurIPS 2025 · similarity 0.87)
- AffineQuant: Affine Transformation Quantization for Large Language Models (ICLR 2024 · similarity 0.87)
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (ICLR 2024 · similarity 0.87)
- LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.87)
- Matryoshka Quantization (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.