Research map / Architectures and efficiency / Quantization
ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ParetoQ, a unified QAT framework that jointly tunes training budget, quantization grid/range functions, and bit-width across 1 to 4 bits, showing 2-bit and ternary models lie on the accuracy-size Pareto frontier.
Problem
Conflicting conclusions on the optimal bit-width for LLM quantization (4-bit vs 1.58-bit) arise because prior low-bit scaling-law studies used inconsistent training schemes and quantization functions across bit widths.
Approach
A unified framework that fixes training schedule (full-precision pretraining followed by QAT finetuning with ~10% budget) and a bit-specific quantizer (Elastic Binarization for 1-bit, SEQ for 1.58/2-bit, LSQ for 3/4-bit) so that bit widths can be compared apples-to-apples, with learnable range scales and stretched elastic quantization grids for sub-4-bit.
Key results
- On LLaMA-3 8B at 2-bit, ParetoQ reaches 67.0 average vs FP 70.2 (3.4-point gap) and outperforms the best QAT baseline EfficientQAT by 5.7 points.
- A ParetoQ ternary 600M MobileLLM model achieves 55.5 average accuracy, surpassing the 1-bit Era ternary 700M model (48.0) and approaching the 1-bit Era 3B model (54.5).
- At 1.58-bit on LLaMA-3 8B, ParetoQ reaches 69.0 average accuracy and 8.6 Wiki2 perplexity versus 65.6 and 11.7 for 1-bit Era.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QAT finetuning after full-precision pretraining rather than PTQ or QAT from scratch | ParetoQ quantized models · 1-bit, 1.58-bit, 2-bit, 3-bit, and 4-bit quantization | accuracy | Helped |
| Allocate approximately 10% of the training budget to QAT after predominantly full-precision training | ParetoQ training schedule · MobileLLM-125M; 1.58-bit, 2-bit, 3-bit, and 4-bit | accuracy: ~ 90% | Helped |
| Fine-tune pretrained weights instead of training quantized models from scratch | Low-bit quantized networks · Across bit widths | accuracy | Helped |
| Increase finetuning tokens for binary, ternary, and 2-bit quantization compared with 3-bit and 4-bit | ParetoQ QAT · Across bit widths | accuracy: 10B tokens; 30B tokens | Helped |
| Use learnable range settings instead of statistics-based fixed clipping | Low-bit quantization functions · Across bit widths | accuracy | Helped |
| Use SEQ balanced quantization grids instead of LSQ grids | ParetoQ quantization function · Ternary and 2-bit quantization | accuracy | Helped |
| Use LSQ with zero in the output levels instead of SEQ | ParetoQ quantization function · 3-bit and 4-bit quantization | accuracy | Helped |
| Use range clipping instead of min-max quantization | Low-bit quantized models · Below 4-bit quantization | accuracy | Helped |
| Use range clipping instead of min-max quantization | 4-bit quantized models · 4-bit quantization | accuracy | Hurt |
| Increase quantization from 1-bit to 1.58-bit, 2-bit, or 3-bit | ParetoQ · Accuracy-size trade-off | accuracy-size trade-off | Helped |
| Use 2-bit quantization instead of 4-bit quantization | ParetoQ CPU kernels · Accuracy-speed trade-off | accuracy-speed trade-off | Helped |
| Use ParetoQ ternary quantization instead of 1-bit era ternary quantization | LLaMA-3 3B · Ternary zero-shot reasoning | Avg.: 54.5 -> 61.9 | Helped |
Surprising findings
- Binary quantization significantly compromises accuracy, while ternary, 2-bit and 3-bit are roughly tied and often exceed 4-bit in the size-accuracy trade-off.
- Fine-tuning from pretrained full-precision weights improves even binary and ternary models, challenging the claim that ternary LLMs must be trained from scratch.
Limitations
- Speed results are limited to a CPU 2-bit kernel and a limited range of settings, and INT2 tensor-core support is lacking on GPUs.
- Ternary storage benefits are hard to realize in practice due to indexing and packing overheads.
Experimental setup
Datasets: WikiText2 commonsense reasoning tasks (ARC, BoolQ, PIQA, SIQA, HellaSwag, OBQA, WinoGrande)
Benchmarks: ARC-e ARC-c BoolQ PIQA SIQA HellaSwag OBQA WinoGrande WikiText2
Metrics: zero-shot accuracy average accuracy perplexity
Baselines: BiLLM ARB-LLM PB-LLM DB-LLM TernaryLLM 1-bit Era LLM-QAT EfficientQAT GPTQ AWQ OmniQuant SpinQuant
Builds on: LSQ Elastic Binarization BitNet b1.58 EfficientQAT LLM-QAT
Closest papers by meaning
- Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment (NeurIPS 2025 · similarity 0.91)
- PTMQ: Post-training Multi-Bit Quantization of Neural Networks (AAAI 2024 · similarity 0.90)
- QBB: Quantization with Binary Bases for LLMs (NeurIPS 2024 · similarity 0.90)
- CBQ: Cross-Block Quantization for Large Language Models (ICLR 2025 · similarity 0.90)
- LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.89)
- PB-LLM: Partially Binarized Large Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.