Research map / Architectures and efficiency / Quantization
The case for 4-bit precision: k-bit Inference Scaling Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 4-bit rather than other tested precisions at a fixed total model-bit budget | Zero-shot quantized LLMs · LAMBADA, PiQA, HellaSwag, Winogrande | mean zero-shot accuracy | Helped |
| Reduce precision from 4-bit to 3-bit | Zero-shot quantized LLMs · LAMBADA, PiQA, HellaSwag, Winogrande | mean zero-shot accuracy | Hurt |
| Use a smaller quantization block size (ablation) | 4-bit quantized Pythia, GPT-2, BLOOM, BLOOMZ, and OPT · LAMBADA, PiQA, HellaSwag, Winogrande | mean zero-shot accuracy: 0.24 bits per parameter | Helped |
| Use small block sizes for 3-bit quantization (ablation) | 3-bit zero-shot quantized models · LAMBADA, PiQA, HellaSwag, Winogrande | mean zero-shot accuracy | Helped |
| Use small block sizes for 5-bit quantization (ablation) | 5-bit zero-shot quantized models · LAMBADA, PiQA, HellaSwag, Winogrande | mean zero-shot accuracy | Helped |
| Use quantile quantization instead of other tested data types (ablation) | Zero-shot quantized LLMs · LAMBADA, PiQA, HellaSwag, Winogrande | mean zero-shot accuracy | Helped |
| Use Float rather than Integer or Dynamic Exponent data types (ablation) | 4-bit Pythia and other tested models · LAMBADA, PiQA, HellaSwag, Winogrande | mean zero-shot accuracy | Helped |
| Use Integer rather than Float data types at 5-bit precision (ablation) | 5-bit quantized models · LAMBADA, PiQA, HellaSwag, Winogrande | mean zero-shot accuracy | Mixed |
| Add proxy quantization for the 2% most significant outlier dimensions, storing them in 16-bit rather than 3-bit (ablation) | 3-bit OPT and Pythia · LAMBADA, PiQA, HellaSwag, Winogrande | mean zero-shot accuracy: 2%; 16-bit; 3-bit | Helped |
| Add proxy quantization for outlier dimensions (ablation) | 4-bit quantized models · LAMBADA, PiQA, HellaSwag, Winogrande | mean zero-shot accuracy | No clear effect |
| Add distribution centering before weight quantization (ablation) | Weight quantization · All tested scenarios | bit-level scaling | No clear effect |
| Apply different data types or block sizes at 6–8-bit precision (ablation) | 6–8-bit quantized models · LAMBADA, PiQA, HellaSwag, Winogrande | bit-level scaling | No clear effect |
Closest papers by meaning
- Scaling Laws for Floating–Point Quantization Training (ICML 2025 · similarity 0.90)
- ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs (NeurIPS 2026 · similarity 0.89)
- RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remapping (NeurIPS 2026 · similarity 0.89)
- ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization (NeurIPS 2025 · similarity 0.88)
- KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.88)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.