Research map / Architectures and efficiency / Quantization
BitNet: 1-bit Pre-training for Large Language Models
Hongyu Wang, Shuming Ma, Lingxiao Ma, Lei Wang, Wenhui Wang, Li Dong, Shaohan Huang, Huaijie Wang, Jilong Xue, Ruiping Wang, Yi Wu, Furu Wei
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train 1-bit weights from scratch (BitNet b1) instead of post-training W1A8 absmax quantization | FP16 Transformer 6.7B · Zero-shot average, 4 tasks | average accuracy: 44.6 -> 55.9 | Improved |
| add SubLN instead of Pre-LN (ablation) | BitNet b1 (absmax) · Ablation, 1-bit weights from scratch | perplexity: 22.11 -> 20.34 | Improved |
| add SubLN instead of BMT (ablation) | BitNet b1 (absmax) · Ablation, 1-bit weights from scratch | perplexity: 22.98 -> 20.34 | Improved |
| use absmax activation quantization instead of elastic function (ablation) | BitNet b1 with Pre-LN · Ablation, 1-bit weights from scratch | perplexity: 24.05 -> 22.11 | Improved |
| use absmax instead of elastic and SubLN instead of Pre-LN (full BitNet) (ablation) | BitNet b1 · Ablation, zero-shot average | average accuracy: 50.2 -> 52.3 | Improved |
| quantize the KV cache to 4 bits (ablation) | BitNet b1.58 3B · Zero-shot average, 7 end tasks | average accuracy: 50.2 -> 50.0 | No clear effect |
| quantize the KV cache to 4 bits (ablation) | BitNet b1.58 3.9B · Zero-shot average, 7 end tasks | average accuracy: 51.2 -> 51.3 | No clear effect |
| scale BitNet b1.58 from 700M to 3B parameters | BitNet b1.58 · Zero-shot average, 7 end tasks | average accuracy: 44.3 -> 50.2 | Improved |
| replace LLaMA LLM with BitNet b1.58 at 700M | LLaMA LLM 700M · RedPajama perplexity | perplexity: 12.33 -> 12.87 | Hurt |
| replace LLaMA LLM with BitNet b1.58 at 3B | LLaMA LLM 3B · RedPajama perplexity | perplexity: 10.04 -> 9.91 | Improved |
| replace FP16 LLaMA with BitNet b1.58 for decoding | LLaMA LLM 3B · Decoding latency | latency (ms): 5.07 -> 1.87 | Improved |
| replace FP16 LLaMA with BitNet b1.58 for serving 70B | LLaMA LLM 70B · Throughput, two A100 80GB | throughput (tokens/s): 333 -> 2977 | Improved |
Closest papers by meaning
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.91)
- OneBit: Towards Extremely Low-bit Large Language Models (NeurIPS 2024 · similarity 0.91)
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.90)
- ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization (NeurIPS 2024 · similarity 0.90)
- QuEST: Stable Training of LLMs with 1-Bit Weights and Activations (ICML 2025 · similarity 0.90)
- LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
