Research map / Architectures and efficiency / Quantization
BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BitStack, a training-free weight compression that iteratively decomposes LLM weights into ~1-bit residual blocks (absolute value decomposition with activation-aware scaling) sorted into a universal stack, enabling megabyte-level memory-performance trade-offs.
Problem
Deploying LLMs on local devices with variable, limited RAM; existing compression (quantization, pruning, distillation) fixes a compression ratio and requires separate compression runs per setting, while prior weight-decomposition methods collapse at high compression ratios.
Approach
Decompose each weight matrix iteratively via SVD on the absolute-value matrix while keeping the sign matrix (~1 bit/parameter per residual block), apply activation-aware row scaling, and sort all residual blocks across layers into one universal stack so any prefix of blocks gives a usable model at arbitrary memory size.
Key results
- On Llama 3.1 70B at ~79% compression, BitStack reaches 75.5 zero-shot avg vs. best baseline roughly 34 (2-bit baselines fail), retaining about 89% of FP16 performance at extreme ratios.
- At the lowest compression ratio (64%), BitStack Llama 3.1 8B recovers about 96% and 70B about 98% of FP16 zero-shot performance, trailing the best baselines by 1.7 and 0.8 points.
- Ablations show activation-aware scaling adds about 8 points of zero-shot accuracy at 4000MB, and replacing AVD with vanilla SVD collapses performance until 12000MB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add activation-aware scaling (ablation) | BitStack · Llama 3.1 8B at 4000 MB | Avg. zero-shot performance: 8-point improvement | Helped |
| Use absolute value decomposition instead of vanilla SVD (ablation) | BitStack · Llama 3.1 8B across compression ratios | Avg. zero-shot performance | Helped |
| Add activation-aware scaling and absolute value decomposition (ablation) | BitStack · Llama 3.1 8B across compression ratios | Avg. zero-shot performance | Helped |
| Use Average sorting rather than Random or Greedy sorting (ablation) | BitStack · Llama 3.1 8B at 4000 MB | Avg. zero-shot performance: 16 points | Helped |
| Use Average sorting rather than Random or Greedy sorting (ablation) | BitStack · Llama 3.1 8B at 4500 MB | Avg. zero-shot performance: 16 points | Helped |
| Use Average sorting rather than Random or Greedy sorting (ablation) | BitStack · Llama 3.1 8B at 5000 MB | Avg. zero-shot performance: 7 points | Helped |
| Use a calibration set size of 64 (ablation) | BitStack · Llama 3.1 8B at 4000 MB | Avg. zero-shot performance | Helped |
| Increase the number of kept singular vectors beyond 1 (ablation) | BitStack · Llama 3.1 8B at fixed memory footprints | Avg. zero-shot performance | No clear effect |
| Keep only the largest singular value and its vectors (ablation) | BitStack · Llama 3.1 8B across memory footprints | Avg. zero-shot performance | Hurt |
Surprising findings
- Vanilla SVD-based decomposition collapses across compression ratios, while absolute value decomposition keeps performance usable at high compression.
- GPTQ on Llama 3 and 3.1 models is unstable and sometimes collapses, with perplexity in the hundreds of thousands at 2-3 bit.
Limitations
- Gap to quantization baselines remains at lower compression ratios (e.g., 64%), where quantization is competitive.
- Sorting requires perplexity evaluation of each residual block, and scaling depends on a calibration set.
- Evaluation focuses on Llama family models and WikiText2/zero-shot tasks.
Experimental setup
Datasets: WikiText2
Benchmarks: WikiText2 PIQA HellaSwag WinoGrande ARC-easy ARC-challenge LAMBADA MT-Bench
Metrics: perplexity zero-shot accuracy average accuracy MT-Bench score
Baselines: GPTQ AWQ FWSVD ASVD SVD-LLM
Builds on: SVD-LLM AWQ BitNet
Closest papers by meaning
- LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.90)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.90)
- Extreme Compression of Large Language Models via Additive Quantization (ICML 2024 · similarity 0.90)
- PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression (NeurIPS 2024 · similarity 0.90)
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.90)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.