Research map / Architectures and efficiency / Quantization
SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SKIM, a post-training weight-only quantization method that combines greedy channel-wise mixed-precision bit allocation, weighted K-means clustering, and a trainable scaling vector optimized by iterative grouping/scaling, enabling any-bit (including non-integer) quantization.
Problem
Post-training weight-only LLM quantization suffers large perplexity drops at low bit widths, and existing methods offer only a few fixed bit levels that often require manual tuning and extra memory.
Approach
Allocates bits across weight channels with a greedy algorithm driven by per-channel quantization error, clusters each channel with sensitivity-weighted K-means at its allocated bit width, and adds a trainable per-column scaling vector optimized by alternating fixed-label K-means and gradient descent on the scale, so that any average bit width (including non-integer) is supported.
Key results
- On LLaMA-7B at 3-bit, SKIM reports WikiText2 perplexity 6.21 vs SqueezeLLM 6.32 and OmniQuant 6.48 (FP16 5.68).
- At 2.25 bits on LLaMA-7B, SKIM reaches WikiText2 perplexity 8.99 vs OmniQuant 9.72 and SqueezeLLM 11.32.
- Abstract claims the gap between 3-bit LLaMA and full precision is narrowed by 16.3% on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add channel-wise greedy mixed-precision bit allocation (ablation) | SKIM · OPT models, perplexity evaluation | perplexity | Hurt |
| add the trainable scaling vector after grouping (ablation) | SKIM · LLaMA quantization | perplexity | Helped |
| use S-diag instead of L-diag for weighted K-means (ablation) | SKIM weighted K-means clustering · 3-bit quantization | perplexity: 6.36 -> 6.33 | Helped |
| use L-full instead of S-diag for scaling-vector loss (ablation) | SKIM scaling-vector optimization · 3-bit quantization | perplexity: 6.27 -> 6.24 | Helped |
| use L-full instead of L-diag for scaling-vector loss (ablation) | SKIM scaling-vector optimization · 3-bit quantization | perplexity: 6.29 -> 6.24 | Helped |
| use S-diag instead of L-diag for scaling-vector loss (ablation) | SKIM scaling-vector optimization · 3-bit quantization | perplexity: 6.29 -> 6.27 | Helped |
| increase iterative optimization beyond the first iteration (ablation) | SKIM · LLaMA quantization | perplexity | Mixed |
| use SKIM instead of SqueezeLLM at 3-bit quantization | LLaMA-7B · MMLU zero-shot and five-shot | average accuracy: 27.2% -> 28.3%; 31.2% -> 33.2% | Helped |
Surprising findings
- Mixed precision increased perplexity on OPT models despite reducing quantization error, which the authors attribute to overfitting on calibration data.
- Empirical SmoothQuant-style scaling values fail in the channel-wise weight setting, increasing perplexity.
Limitations
- Mixed precision was disabled at integer bit levels on OPT models due to calibration overfitting.
- Quantization requires calibration data and per-layer error pre-recording, adding setup cost (about half an hour for LLaMA-7B errors).
Experimental setup
Datasets: C4
Benchmarks: WikiText2 C4 MMLU
Metrics: perplexity accuracy peak memory usage
Baselines: SqueezeLLM OmniQuant DecoupleQ ABQ-LLM
Builds on: SqueezeLLM OmniQuant SmoothQuant
Closest papers by meaning
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.87)
- Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment (NeurIPS 2025 · similarity 0.87)
- GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling (NeurIPS 2026 · similarity 0.87)
- Treasures in Discarded Weights for LLM Quantization (AAAI 2025 · similarity 0.87)
- SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models (ICML 2025 · similarity 0.87)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.