Research map / Architectures and efficiency / Quantization
QBB: Quantization with Binary Bases for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QBB, a post-training weight-only quantization that decomposes LLM weights into N=4 binary matrices with fp16 scales (analytical residual init, iterative gradient refinement) and calibrates scales with data-free student-teacher distillation.
Problem
Low-bit post-training quantization of LLMs, especially sub-7B models, degrades sharply below 4 bits, and existing methods still rely on mixed-precision multiplications that have limited hardware support.
Approach
Approximate each weight matrix as a sum of N binary matrices with fp16 scaling vectors, initialized by cascaded analytical residual binarization and refined layer-wise by block-coordinate gradient descent; then calibrate only the scales with data-free logit and feature distillation on self-generated text, so inference needs almost only summations.
Key results
- On LLaMA-2 7B, QBB (W(4x1)A16g128) reaches 5.21 WikiText2 perplexity versus 5.02 for OmniQuant W4A16g128-level baselines and 5.12 FP16; on LLaMA-2 13B it reaches 4.63 vs 4.57 FP16.
- On Phi-2 (2.7B), the full method reduces perplexity from 12.32 (layer-wise binarization only) to 10.48 with holistic distillation, showing data-free calibration matters most for small models.
- Ablation: cascaded residual initialization gives roughly 2x smaller per-layer reconstruction error than global-error initialization on Phi-2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add data-free holistic distillation to input-agnostic binarization (ablation) | QBB · WikiText2, Phi-2 2.7B | perplexity: 12.32 -> 10.48 | Helped |
| add data-free holistic distillation to input-agnostic binarization (ablation) | QBB · WikiText2, LLaMA-2 7B | perplexity: 5.37 -> 5.21 | Helped |
| use KD-MSE with gradual teacher/student block swapping instead of CE, KD, or KD-MSE (ablation) | QBB calibration on Phi-2 2.7B · WikiText2 | perplexity: 11.82 -> 10.40; 10.61 -> 10.40; 10.48 -> 10.40 | Helped |
| initialize binary bases from FP16 weights rather than GPTQ or OmniQuant int4 weights (ablation) | QBB on LLaMA-2 7B · WikiText2 | perplexity: 5.49 -> 7.10; 5.21 -> 7.10 | Hurt |
| initialize binary bases from OmniQuant rather than GPTQ int4 weights (ablation) | QBB on LLaMA-2 7B · WikiText2 | perplexity: 5.49 -> 5.21 | Helped |
| use filtered generated samples with the highest teacher-student discrepancy (ablation) | QBB calibration on Phi-2 2.7B · WikiText2 | perplexity | Helped |
| use cascaded residual initialization rather than direct global-error initialization (ablation) | QBB binary bases · Phi-2 2.7B layer reconstruction | L2 reconstruction error: 2× smaller | Helped |
| use iterative one-binary-weight-at-a-time optimization rather than joint optimization (ablation) | QBB binary bases · Phi-2 2.7B layer reconstruction | average reconstruction error: one order of magnitude higher | Helped |
Surprising findings
- Directly binarizing fp16 weights without progressive quantization (starting from GPTQ/OmniQuant int4 weights) degrades results, so the starting point matters.
- Only the first and last binary bases change little during optimization (0.01-0.2% bit flips), while the second and third change about 40-50%.
Limitations
- Efficiency gains are theoretical; no custom hardware or kernel implementation was built.
- Inherits biases from the original pre-trained models.
- Evaluation focuses on WikiText2 perplexity rather than downstream task accuracy.
Experimental setup
Datasets: self-generated synthetic text from the teacher model
Benchmarks: WikiText2
Metrics: perplexity
Baselines: RTN GPTQ AWQ OmniQuant SpQR QuIP#
Builds on: BinaryConnect XNOR-Net GPTQ OmniQuant Data-free quantization (Liu et al. 2023)
Closest papers by meaning
- ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization (NeurIPS 2025 · similarity 0.90)
- Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment (NeurIPS 2025 · similarity 0.90)
- PB-LLM: Partially Binarized Large Language Models (ICLR 2024 · similarity 0.90)
- GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling (NeurIPS 2026 · similarity 0.89)
- Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression (NeurIPS 2025 · similarity 0.88)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.