Research map / Architectures and efficiency / Quantization
LittleBit: Ultra Low-Bit Quantization via Latent Factorization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LittleBit, a low-rank latent factorization with binarized factors, multi-scale (row/column/latent) FP16 scales, Dual-SVID initialization, and residual compensation, trained via QAT with KD to reach 0.1 to 1.0 BPW on OPT, Llama, Llama2, Llama3, Phi-4 and QwQ.
Problem
LLM weight quantization degrades sharply in the sub-1-bit regime (below ~1 bit per weight), where existing PTQ methods such as STBLLM lose most of their accuracy.
Approach
Factor each weight matrix as a low-rank product U V^T, binarize the factors, and learn FP16 row, column and latent scales; initialize with Dual-SVID and add a parallel binarized residual path that models the initial approximation error, then train with QAT and knowledge distillation.
Key results
- At 0.55 BPW on Llama2-7B, LittleBit reaches WikiText-2 perplexity 10.47 versus 30.67 for STBLLM at the same BPW.
- At 0.1 BPW on Llama2-13B, memory falls to 0.84 GB (31.02x vs FP16) with perplexity 15.09.
- A custom 1-bit GEMV CUDA kernel reaches up to 11.6x speedup over an optimized FP16 baseline at 0.1 BPW on a Llama2-70B MLP layer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LittleBit rather than STBLLM at 0.55 BPW | Llama2-7B · WikiText-2 | perplexity: 30.67 -> 10.47 | Helped |
| Use LittleBit rather than STBLLM at 0.8 BPW | Llama2-13B · WikiText-2 | perplexity: 11.90 -> 8.52 | Helped |
| Use LittleBit rather than STBLLM at 0.55 BPW | Llama3-8B · WikiText-2 | perplexity: 241.95 -> 18.47 | Helped |
| Use LittleBit rather than STBLLM at 0.3 BPW | Llama2-7B · WikiText-2 | perplexity: 1.8e3 -> 12.00 | Helped |
| Use LittleBit rather than STBLLM at 0.55 BPW | Llama2-7B · Common-sense reasoning | average accuracy: 44.29 -> 47.26 | Helped |
| Use LittleBit rather than STBLLM at 0.3 BPW | Llama2-7B · Common-sense reasoning | average accuracy: 35.78 -> 45.20 | Helped |
| Add Residual Compensation (ablation) | LittleBit on OPT-1.3B at 0.1 BPW · WikiText-2 | perplexity: 48.512 -> 60.011 | Hurt |
| Add Residual Compensation (ablation) | LittleBit on OPT-1.3B at 0.3 BPW · WikiText-2 | perplexity: 29.724 -> 28.984 | Helped |
| Add Residual Compensation (ablation) | LittleBit on OPT-1.3B at 0.55 BPW · WikiText-2 | perplexity: 24.788 -> 23.457 | Helped |
| Replace STE with SmoothSign (ablation) | LittleBit on OPT-1.3B at 0.1 BPW · WikiText-2 | perplexity: 60.401 -> 60.011 | Helped |
| Replace STE with SmoothSign (ablation) | LittleBit on OPT-1.3B at 1.00 BPW · WikiText-2 | perplexity: 20.322 -> 20.329 | No clear effect |
| Increase K/V latent rank to 4× (ablation) | LittleBit on Llama3-8B · WikiText-2, approximately 0.1 BPW | perplexity: 25.80 -> 24.93 | Helped |
Surprising findings
- Residual compensation hurt OPT-1.3B at 0.1 BPW (PPL 60.011 with vs 48.512 without), contrary to its benefit at other bit widths.
- SVD-based compression outperformed Wanda pruning at high compression ratios (25% retention: SVD 52.6 vs pruning 1842.7 PPL on Llama-7B).
Limitations
- Quantization cliff between 0.3 and 0.1 BPW; 0.1 BPW perplexity degrades substantially (e.g., Phi-4 and QwQ-32B), and generated samples at 0.1 BPW show hallucination.
- Reasoning results on Phi-4 and QwQ-32B are shown only for 7 zero-shot tasks and LittleBit trails STBLLM at 0.55 BPW on Phi-4.
- Measured speedups are modest in practice due to memory-bound small-batch inference; the custom kernel is not optimized to industry libraries.
Experimental setup
Datasets: WikiText-2 C4
Benchmarks: WikiText-2 C4 PTB WinoGrande OpenBookQA HellaSwag BoolQ ARC-Easy ARC-Challenge PIQA
Metrics: perplexity zero-shot accuracy memory footprint (GB) speedup
Baselines: STBLLM BiLLM OneBit BinaryMoS GPTQ OmniQuant
Builds on: SVD OneBit BiLLM Straight-Through Estimator SVD-LLM
Closest papers by meaning
- OneBit: Towards Extremely Low-bit Large Language Models (NeurIPS 2024 · similarity 0.93)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.92)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.91)
- PB-LLM: Partially Binarized Large Language Models (ICLR 2024 · similarity 0.91)
- PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression (NeurIPS 2024 · similarity 0.91)
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.