Research map / Architectures and efficiency / Quantization
OneBit: Towards Extremely Low-bit Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Quantizes LLM linear-layer weights to 1 bit (sign matrix plus two FP16 value vectors) and initializes the 1-bit model via sign-value-independent decomposition, then recovers capability with quantization-aware knowledge distillation.
Problem
Post-training and existing QAT quantization degrade severely below 2-3 bits, so LLMs have not been usable at extremely low (1-bit) weight precision when quantizing pretrained models.
Approach
Represent each weight matrix as a ±1 sign matrix times two FP16 value vectors inside a new Linear layer (with Post-LayerNorm), initialize these from a rank-1 SVID (NMF/SVD of |W|), and train with logit cross-entropy plus hidden-state MSE distillation from the FP16 teacher.
Key results
- On LLaMA-7B, OneBit (W1A16) reaches 10.19 Wiki2 perplexity and 51.33 average zero-shot accuracy versus 5.68 and 64.06 for FP16, outperforming OmniQuant (W2A16) at 48.17.
- On LLaMA-13B, OneBit achieves 55.17 average zero-shot accuracy, above OmniQuant's 52.51 at 2-bit, retaining about 81% of FP16 performance.
- Quantized LLaMA-13B size is about 2.2 GB versus 26.0 GB for FP16 (91.5% compression ratio per the paper's table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OneBit W1A16 instead of OmniQuant W2A16 | OPT-1.3B · WikiText2 | perplexity: 42.43 -> 25.42 | Helped |
| Use OneBit W1A16 instead of OmniQuant W2A16 | LLaMA-7B · Zero-shot average | accuracy: 48.17 -> 51.33 | Helped |
| Add hidden-state MSE alignment to KD loss (ablation) | OneBit, LLaMA-7B · WikiText2 | perplexity: 13.48 -> 10.19 | Helped |
| Set hidden-state MSE weight α=10 rather than α=1 (ablation) | OneBit, LLaMA-7B · Zero-shot average | accuracy: 51.33 -> 52.05 | Helped |
| Add attention-score alignment to KD + MSE (ablation) | OneBit, LLaMA-7B · WikiText2 | perplexity | Hurt |
| Use NMF rather than SVD for SVID initialization (ablation) | OneBit · Training convergence | convergence speed | Helped |
| Use Post-LayerNorm rather than Pre-LayerNorm (ablation) | OneBit · Quantization-aware training | training stability | Helped |
| Include two FP16 value vectors rather than the BitNet-style architecture without them (ablation) | OneBit · Knowledge transfer training | training stability | Helped |
Surprising findings
- GPTQ and LLM-QAT collapse to near-random perplexity (thousands) at 2-bit on these LLaMA and OPT models.
- BitNet's from-scratch 1-bit architecture is unstable when used for distillation from a pretrained LLaMA-7B.
Limitations
- Performance loss relative to FP16 remains substantial, especially on some zero-shot tasks.
- Mathematical principles for optimal 1-bit parameters are not understood, so capability transfer requires costly KD.
- Method does not naturally extend to higher bit-widths and activation quantization is not addressed.
Experimental setup
Datasets: self-generated corpus from LLaMA-7B (132k entries) C4 WikiText2 alpaca_en_52k MMLU
Benchmarks: WikiText2 C4 Winogrande HellaSwag PIQA BoolQ ARC-easy ARC-challenge MMLU
Metrics: perplexity zero-shot accuracy average accuracy memory footprint
Baselines: GPTQ LLM-QAT OmniQuant BitNet Pythia-1.0B TinyLLaMA-1.1B LowRank LLaMA
Builds on: BitNet LLM-QAT OmniQuant GPTQ
Closest papers by meaning
- LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.93)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.91)
- PB-LLM: Partially Binarized Large Language Models (ICLR 2024 · similarity 0.91)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.91)
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.91)
- ARB-LLM: Alternating Refined Binarizations for Large Language Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.