Research map / Large language models / Language, safety and interpretability
PB-LLM: Partially Binarized Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PB-LLM, which binarizes most LLM weights while keeping a small fraction of salient weights at higher bit-width, and recovers accuracy via a Hessian-guided PTQ (PB-GPTQ) and a QAT scheme with frozen salient weights and analytically optimal scaling factors.
Problem
Network binarization of LLM weights to one bit collapses model performance, so extreme low-bit LLM compression that preserves reasoning ability is unsolved.
Approach
Partially binarize the weight matrix: detect a small fraction of salient weights (by magnitude) and store them in higher bits, binarize the rest with an analytically optimal per-channel scale (mean absolute value), and recover performance with GPTQ-style Hessian compensation (PTQ) or QAT with frozen salient weights.
Key results
- On LLaMA-7B at 4-bit-equivalent setting, PB-LLM 30% reaches 66.9 average zero-shot accuracy vs. 68.7 for FP and 66.6 for LLM-QAT (trained 10K vs 100K iterations).
- PB-GPTQ with Hessian criterion on OPT-1.3B reduces C4 perplexity from RTN's thousands to 17.76 at 50% salient weights; on LLaMA-7B PB-GPTQ 50% gives C4 perplexity 8.15 vs GPTQ 4-bit 8.70.
- Existing binarization methods (BNN, XNOR, Bi-Real, ReCU, FDA) applied to OPT-1.3B perform at or below random guessing (mean about 0.30-0.32 vs random 0.36).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply PB-LLM with 30% salient weights instead of LLM-QAT | LLaMA-7B · Seven common-sense tasks average | accuracy: 66.6 -> 66.9 | Helped |
| Preserve 30% salient weights and fine-tune rather than use PB-GPTQ 30% | LLaMA-7B · Seven common-sense tasks average | accuracy: 54.2 -> 66.9 | Helped |
| Reduce the preserved salient-weight fraction from 30% to 10% (ablation) | PB-LLM · Seven common-sense tasks average | accuracy: 66.9 -> 60.6 | Hurt |
| Use Hessian rather than magnitude to select salient weights (ablation) | PB-GPTQ · C4 | perplexity: 895.0322 -> 165.6767 | Helped |
| Use group-wise quantization with g=128 rather than non-group-wise quantization (ablation) | PB-GPTQ Magnitude, OPT-1.3B · C4 | perplexity: 895.0322 -> 1230.8537 | Hurt |
| Use group-wise quantization with g=128 rather than non-group-wise quantization (ablation) | PB-GPTQ Hessian, OPT-1.3B · C4 | perplexity: 165.6767 -> 157.8825 | Helped |
| Use group-wise rather than non-group-wise PB-GPTQ (ablation) | PB-GPTQ · C4 | perplexity: The group-wise PB-GPTQ performs better or worse than the non-group-wise PB-GPTQ, but the difference is not significant. | Mixed |
| Use PB-GPTQ rather than RTN | OPT-1.3B with 10% salient weights by magnitude · C4 | perplexity: 4889.0385 -> 895.0322 | Helped |
| Preserve 10% salient weights rather than 30% with PB-GPTQ (ablation) | LLaMA-7B · Seven common-sense tasks average | accuracy: 54.2 -> 36.5 | Hurt |
| Use PB-GPTQ rather than GPTQ 4b | LLaMA-7B with 50% salient weights · C4 | perplexity: 8.6977 -> 8.1466 | Helped |
| Use PB-GPTQ rather than GPTQ 4b | LLaMA-7B with 50% salient weights · WikiText2 | perplexity: 8.1368 -> 6.3089 | Helped |
| Use PB-GPTQ rather than GPTQ 4b | LLaMA-7B with 50% salient weights · PTB | perplexity: 57.9951 -> 54.8674 | Helped |
Surprising findings
- Existing binarization algorithms collapse LLMs to below random-guess performance on common sense reasoning.
- Freezing salient weights and applying optimal scaling without any retraining already retains some language capacity in partially binarized OPT-1.3B.
Limitations
- Storage cost estimate uses an uncompressed bitmap index and ignores index savings; actual bit-width is an upper bound.
- PTQ performance drops sharply at low salient fractions (e.g., PB-GPTQ 10% gives 36.5 average on LLaMA-7B), and QAT evaluation is limited to LLaMA-7B and OPT-1.3B.
- Methodology exploration is mostly on OPT-1.3B due to compute constraints; training uses a small 1B-token subset.
Experimental setup
Datasets: RedPajama-simple-1B C4 WikiText2 PTB BoolQ PIQA HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenBookQA
Benchmarks: BoolQ PIQA HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenBookQA WikiText2 C4 PTB
Metrics: accuracy average accuracy over seven tasks perplexity
Baselines: RTN SmoothQuant LLM-QAT GPTQ SparseGPT BNN XNOR Bi-Real ReCU FDA
Builds on: GPTQ SparseGPT XNOR-Net BiBench AWQ
Closest papers by meaning
- ARB-LLM: Alternating Refined Binarizations for Large Language Models (ICLR 2025 · similarity 0.95)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.94)
- LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.91)
- OneBit: Towards Extremely Low-bit Large Language Models (NeurIPS 2024 · similarity 0.91)
- Treasures in Discarded Weights for LLM Quantization (AAAI 2025 · similarity 0.91)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.