Research map / Architectures and efficiency / Quantization
SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SliM-LLM, a group-wise mixed-precision post-training quantization method for LLM weights that allocates bit-widths per group by salience (SBA) and calibrates quantizers with salience weighting (SQC), with an extension SliM-LLM+ that adds gradient-based OmniQuant calibration.
Problem
Low-bit (≤3-bit) post-training weight quantization of LLMs suffers severe accuracy loss, while existing mixed-precision methods are element-wise and hardware-unfriendly due to bitmaps or codebook indices.
Approach
Salient weights cluster in structured channels due to activation outliers, so bit-widths can be allocated at the group level by average group salience (searching a KL-divergence-optimal ratio of low/high-bit groups with balanced counts), and per-group quantizer scale/zero are calibrated with a salience-aware search over a clipping factor tau, keeping storage group-structured.
Key results
- On 2-bit LLaMA-13B with GPTQ backbone, WikiText2 perplexity drops from 20.44 to 8.87 (SliM-LLM), and 2-bit LLaMA2-13B from 28.14 to 9.41.
- On 2-bit LLaMA-7B, SliM-LLM reaches perplexity 14.58 versus GPTQ's 152.31 on WikiText2, and reports 2-bit LLaMA-7B zero-shot average 43.84 vs GPTQ 39.65.
- SliM-LLM+ at 2-bit on LLaMA-7B reaches WikiText2 perplexity 9.68 vs OmniQuant 9.72 (LLaMA-1-7B) and 10.87 vs 11.06 on LLaMA-2-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add group-wise salience-determined bit allocation with SQC to GPTQ | GPTQ on LLaMA-1 13B and LLaMA-2 13B · WikiText2, 2-bit | perplexity: 20.44 -> 8.87; 28.14 -> 9.41 | Helped |
| Add SliM-LLM group-wise mixed-precision quantization | LLaMA-7B · Zero-shot average accuracy, 2-bit | accuracy: 39.65 -> 43.84 | Helped |
| Add SBA to OmniQuant with a learnable quantizer | LLaMA-7B · Zero-shot average accuracy, 2-bit | accuracy: 49.11 -> 51.02 | Helped |
| Use salience-determined bit allocation instead of ILP allocation (ablation) | LLaMA-7B · WikiText2, 2-bit | perplexity: 17.55 -> 14.58 | Helped |
| Use KL divergence rather than MSE as the SBA objective (ablation) | SBA on LLaMA-7B · WikiText2, 2-bit | perplexity: 21.94 -> 14.58 | Helped |
| Add salience-weighted quantizer calibration (ablation) | OPT-1.3B quantization · Selected channel | absolute weight quantization error: 0.0055 -> 0.0039 | Helped |
| Use random or head-tail group bit allocation instead of SBA (ablation) | SliM-LLM · OPT models, mixed-precision allocation | perplexity | Hurt |
| Reduce group size from 128 to 64 or 32 (ablation) | SliM-LLM on OPT-6.7B · WikiText2, 2-bit | perplexity: 14.41 -> 13.95; 14.41 -> 12.47 | Helped |
Surprising findings
- Random and head-tail group bit allocation can degrade performance relative to uniform precision at constant average bit-width.
- On LLaMA-3-8B, 2-bit SliM-LLM perplexity is 39.66 while 3-bit improves to 7.16, indicating strong sensitivity to bit-width.
Limitations
- Current GPU deployment tools do not efficiently support mixed-precision 1/2/3-bit inference; 2-bit SliM-LLM throughput on 7B drops to 61.2 tokens/s versus 83.9 for GPTQ.
- Mixed-precision support on GPUs remains limited, reducing inference advantages.
Experimental setup
Datasets: WikiText2
Benchmarks: WikiText2 C4 PIQA ARC-e ARC-c BoolQ HellaSwag Winogrande
Metrics: perplexity zero-shot accuracy memory usage tokens per second
Baselines: GPTQ AWQ OmniQuant AffineQuant PB-LLM QuIP LLM-MQ APTQ RTN
Builds on: GPTQ OmniQuant SparseGPT HAWQ v2
Closest papers by meaning
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.92)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.92)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.92)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.92)
- LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.91)
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.