Research map / Architectures and efficiency / Quantization
SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SLiM, a one-shot LLM compression pipeline combining SLiM-Quant (probabilistic uniform quantization), Wanda pruning to 2:4 or unstructured sparsity, and saliency-based low-rank adapters, with an optional PEFT recovery step.
Problem
One-shot compression of LLMs typically loses substantial accuracy when combining sparsity (especially semi-structured 2:4) with low-bit quantization, and retraining-based recovery is expensive.
Approach
Quantize uniformly with a data-driven MSE-minimizing clipping scale (SLiM-Quant), prune with Wanda, then compute low-rank adapters in closed form via SVD of a saliency-weighted compression error, where the saliency diag(x) is invertible and additive so adapters can be recovered exactly without iterative optimization.
Key results
- SLiM-LoRA improves average zero-shot accuracy of LLaMA-2-7B by up to 5.66% over prior methods under 2:4 sparsity with 4-bit weights.
- Models compressed with SLiM achieve up to 4.3x and 3.8x layer-wise speedup on RTX 3060 and A100 GPUs.
- Optional PEFT on adapters improves LLaMA-2-13B accuracy by up to 1.66% over SLiM without fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use saliency-based adapters instead of norm-based Naive-LoRA adapters (ablation) | SLiM compression · LLaMA-2-7B, 50% 2:4 sparsity and 4-bit weight quantization | average zero-shot accuracy: 48.45 -> 51.15 | Helped |
| quantize the low-rank adapters (ablation) | SLiM-LoRA · LLaMA-2-7B, 50% 2:4 sparsity and 4-bit weight quantization | average zero-shot accuracy: 51.15 -> 51.02 | Hurt |
| use saliency-based adapters instead of norm-based Naive-LoRA adapters (ablation) | SLiM compression · LLaMA-2-7B, 50% unstructured sparsity and 4-bit weight quantization | average zero-shot accuracy: 51.52 -> 54.26 | Helped |
| quantize the low-rank adapters (ablation) | SLiM-LoRA · LLaMA-2-7B, 50% unstructured sparsity and 4-bit weight quantization | average zero-shot accuracy: 54.26 -> 54.18 | Hurt |
| fine-tune the adapters after compression (ablation) | SLiM-LoRA · LLaMA-2-7B, 50% 2:4 sparsity and 4-bit weight quantization | average zero-shot accuracy: 51.15 -> 52.12 | Helped |
| fine-tune the adapters after compression (ablation) | SLiM-LoRA · LLaMA-2-7B, 50% unstructured sparsity and 4-bit weight quantization | average zero-shot accuracy: 54.26 -> 54.69 | Helped |
| use activation-aware output-error quantization instead of weight-error quantization (ablation) | SLiM-Quant · LLaMA-2-7B, 50% 2:4 sparsity and 4-bit weight quantization | average zero-shot accuracy: 51.15 -> 51.22 | Helped |
| use activation-aware output-error quantization instead of weight-error quantization (ablation) | SLiM-Quant · LLaMA-2-7B, 50% unstructured sparsity and 4-bit weight quantization | average zero-shot accuracy: 54.26 -> 54.46 | Helped |
| add 8-bit input quantization (ablation) | SLiM-LoRA · LLaMA-2-7B, 50% 2:4 sparsity and 4-bit weight quantization | average zero-shot accuracy: 51.15 -> 50.99 | Hurt |
Surprising findings
- Under 50% unstructured sparsity with 4-bit weights, SLiM-LoRA on LLaMA-2-7B reaches 54.26 average accuracy vs dense 56.6, and in some OPT sizes compressed models exceed dense accuracy at equal parameter size.
- Output-error-aware SLiM-Quant^O gives only a small gain over weight-error SLiM-Quant^W.
Limitations
- Low-rank adapters add full-precision or 4-bit overhead that partially offsets memory and compute savings.
- Uniform SLiM-Quant alone performs poorly without low-rank compensation (e.g., LLaMA-2-7B 31.11 average accuracy in quantization-only setting).
- Output-error variant SLiM-Quant^O adds irregular memory access overhead.
Experimental setup
Datasets: C4 WikiText2
Benchmarks: MMLU PIQA ARC-Easy ARC-Challenge WinoGrande OpenBookQA WikiText2 perplexity
Metrics: zero-shot accuracy perplexity layer-wise speedup memory reduction
Baselines: Wanda SparseGPT Magnitude Pruning OPTQ OmniQuant AffineQuant AWQ L2QER JSQ MaskLLM
Builds on: Wanda AWQ L2QER SparseGPT Optimal Brain Surgeon
Closest papers by meaning
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.93)
- SqueezeLLM: Dense-and-Sparse Quantization (ICML 2024 · similarity 0.89)
- LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.89)
- Effective Interplay between Sparsity and Quantization: From Theory to Practice (ICLR 2025 · similarity 0.88)
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs (NeurIPS 2025 · similarity 0.88)
- SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.