Research map / Architectures and efficiency / Quantization
Matryoshka Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Matryoshka Quantization (MatQuant), a multi-scale training method that jointly optimizes int8, int4 and int2 precisions by exploiting nested MSB slicing of integer weights, applied on top of OmniQuant and QAT for Gemma-2 and Mistral LLMs.
Problem
Quantizing LLM weights to low bit-widths such as int2 severely degrades quality, and current methods produce separate models per precision rather than one model that can serve multiple precisions.
Approach
Exploit the nested (Matryoshka) bit structure of integer types: slice the most significant bits of an int8-quantized weight to obtain int4 and int2 models, and jointly optimize the loss at each precision during training, so the shared weights serve all bit-widths; this also yields interpolated int6/int3 models and layer-wise Mix'n'Match.
Key results
- With OmniQuant, MatQuant's int2 models beat the int2 baseline by 1.04%, 3.11%, and 3.01% average accuracy on Gemma-2 2B, 9B, and Mistral 7B, while int8 and int4 stay within about 0.5% of baselines.
- With QAT, int2 models outperform the baseline by 4.46%, 6.27%, and 7.02% on Gemma-2 2B, 9B, and Mistral 7B.
- Interpolated int6 and int3 models sliced from the MatQuant model perform comparably to explicitly trained baselines without extra training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly train int8, int4, and int2 instead of training int2 independently | OmniQuant on Gemma-2 2B · int2 FFN | Task Avg.: $51.33$ -> $52.37$ | Helped |
| Jointly train int8, int4, and int2 instead of training int2 independently | OmniQuant on Gemma-2 9B · int2 FFN | Task Avg.: $60.24$ -> $63.35$ | Helped |
| Jointly train int8, int4, and int2 instead of training int2 independently | OmniQuant on Mistral 7B · int2 FFN | Task Avg.: $59.74$ -> $62.75$ | Helped |
| Jointly train int8, int4, and int2 instead of training int2 independently | QAT on Gemma-2 2B · int2 FFN | Task Avg.: $47.74$ -> $52.20$ | Helped |
| Jointly train int8, int4, and int2 instead of training int2 independently | QAT on Gemma-2 9B · int2 FFN | Task Avg.: $56.02$ -> $62.29$ | Helped |
| Jointly train int8, int4, and int2 instead of training int2 independently | QAT on Mistral 7B · int2 FFN | Task Avg.: $54.95$ -> $61.97$ | Helped |
| Jointly train multiple precisions rather than train the given precision independently | OmniQuant and QAT · int8 and int4 FFN across reported models | Task Avg. | No clear effect |
| Remove int8 and int4 loss terms, optimizing only int2 (S.P. Matryoshka Quantization) (ablation) | OmniQuant on Gemma-2 2B, Gemma-2 9B, and Mistral 7B · int2 FFN | Task Avg.: $52.37$ -> $53.42$; $63.35$ -> $64.02$; $62.75$ -> $63.58$ | Helped |
| Remove int8 and int4 loss terms, optimizing only int2 (S.P. Matryoshka Quantization) (ablation) | OmniQuant · int8 and int4 FFN | Task Avg. | Hurt |
| Add int8-to-int2 co-distillation (ablation) | OmniQuant on Gemma-2 9B · int2 FFN | Task Avg.: $63.35$ -> $62.91$ | No clear effect |
| Add int8-to-int2 co-distillation (ablation) | OmniQuant on Gemma-2 9B · int8 and int4 FFN | Task Avg. | No clear effect |
| Allow an extra quantization bucket (E.P. Matryoshka Quantization) (ablation) | OmniQuant on Gemma-2 9B · int2 FFN | Task Avg.: $63.35$ -> $68.25$ | Helped |
Surprising findings
- Slicing an int8 model directly to int2 or int4 degrades badly, while training jointly across precisions makes sliced int4 and int2 much better than sliced int8.
- Adding an extra bucket (an extra bit for outliers, the errata's EP-MatQuant variant) raised Gemma-2 9B int2 accuracy by about 5%, and a Mix'n'Match model at 2.05 bits can match or beat int3.
Limitations
- Experiments are limited to FFN weight-only quantization in most settings; FFN+Attention int2/int3 training with QAT was unstable.
- Floating-point extension is unresolved because exponent slicing yields non-constant bucket sizes.
- The paper's errata reveals an initial bug in the slicing equation; the extra-bucket variant needs custom kernels for sparse extra bits.
Experimental setup
Datasets: C4
Benchmarks: ARC-c ARC-e BoolQ HellaSwag PIQA Winogrande C4 validation perplexity
Metrics: average zero-shot accuracy log perplexity
Baselines: OmniQuant baseline QAT baseline Sliced int8 SpinQuant GPTQ
Builds on: OmniQuant Quantization Aware Training Matryoshka Representation Learning
Closest papers by meaning
- NestQuant: nested lattice quantization for matrix products and LLMs (ICML 2025 · similarity 0.86)
- LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.85)
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (ICLR 2024 · similarity 0.85)
- OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting (ICLR 2025 · similarity 0.85)
- QERA: an Analytical Framework for Quantization Error Reconstruction (ICLR 2025 · similarity 0.85)
- SpinQuant: LLM Quantization with Learned Rotations (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.