AI Paper Atlas

Research map / Architectures and efficiency / Quantization

BitNet: 1-bit Pre-training for Large Language Models

Hongyu Wang, Shuming Ma, Lingxiao Ma, Lei Wang, Wenhui Wang, Li Dong, Shaohan Huang, Huaijie Wang, Jilong Xue, Ruiping Wang, Yi Wu, Furu Wei

JMLR 2025 · Architectures and efficiency › Quantization · approach: Post-training quantization of LLMs

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train 1-bit weights from scratch (BitNet b1) instead of post-training W1A8 absmax quantizationFP16 Transformer 6.7B · Zero-shot average, 4 tasksaverage accuracy: 44.6 -> 55.9Improved
add SubLN instead of Pre-LN (ablation)BitNet b1 (absmax) · Ablation, 1-bit weights from scratchperplexity: 22.11 -> 20.34Improved
add SubLN instead of BMT (ablation)BitNet b1 (absmax) · Ablation, 1-bit weights from scratchperplexity: 22.98 -> 20.34Improved
use absmax activation quantization instead of elastic function (ablation)BitNet b1 with Pre-LN · Ablation, 1-bit weights from scratchperplexity: 24.05 -> 22.11Improved
use absmax instead of elastic and SubLN instead of Pre-LN (full BitNet) (ablation)BitNet b1 · Ablation, zero-shot averageaverage accuracy: 50.2 -> 52.3Improved
quantize the KV cache to 4 bits (ablation)BitNet b1.58 3B · Zero-shot average, 7 end tasksaverage accuracy: 50.2 -> 50.0No clear effect
quantize the KV cache to 4 bits (ablation)BitNet b1.58 3.9B · Zero-shot average, 7 end tasksaverage accuracy: 51.2 -> 51.3No clear effect
scale BitNet b1.58 from 700M to 3B parametersBitNet b1.58 · Zero-shot average, 7 end tasksaverage accuracy: 44.3 -> 50.2Improved
replace LLaMA LLM with BitNet b1.58 at 700MLLaMA LLM 700M · RedPajama perplexityperplexity: 12.33 -> 12.87Hurt
replace LLaMA LLM with BitNet b1.58 at 3BLLaMA LLM 3B · RedPajama perplexityperplexity: 10.04 -> 9.91Improved
replace FP16 LLaMA with BitNet b1.58 for decodingLLaMA LLM 3B · Decoding latencylatency (ms): 5.07 -> 1.87Improved
replace FP16 LLaMA with BitNet b1.58 for serving 70BLLaMA LLM 70B · Throughput, two A100 80GBthroughput (tokens/s): 333 -> 2977Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy