Research map / Architectures and efficiency / Quantization
Effective Interplay between Sparsity and Quantization: From Theory to Practice
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that sparsity and quantization are non-orthogonal, shows sparsity-before-quantization (S→Q) is the better order, and validates the claim empirically on OPT, LLaMA, ViT and ResNet.
Problem
Combining sparsity and quantization is commonly assumed to be orthogonal, adding no error beyond each method alone; the optimal order of applying them and their combined error interaction are not formally understood.
Approach
Formal analysis of max-scaled block-wise quantization and magnitude-based N:M/unstructured sparsity shows S→Q adds no extra tensor-level error while Q→S can, and that dot-product errors compound; an empirical "orthogonality threshold" tests whether combined error exceeds the sum of individual errors.
Key results
- For OPT-125M at 50% sparsity with HBFP6, S→Q gives perplexity 32.51 versus 40.86 for Q→S, while the dense FP32 baseline is 27.65.
- Combined 2:4 sparsity with HBFP6 raises OPT-125M perplexity 6.93x over the dense baseline ratio, exceeding the orthogonality threshold (34.58 vs 34.15 bound) with compounded error.
- Combining sparsity and quantization in the optimal order adds up to 13% additional perplexity error according to the paper's experiments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply magnitude-based sparsity before quantization rather than quantization before sparsity | OPT-125M and LLaMA-2-7B with 50% unstructured or 2:4 sparsity · WikiText2 | perplexity: 34.71 -> 30.22; 45.06 -> 32.76; 10.20 -> 6.51; 18.64 -> 10.68 | Helped |
| Combine sparsity and quantization, even in the S→Q order | OPT-125M, OPT-6.7B, LLaMA-2-7B, LLaMA-3-8B, and ViT-B/16 · WikiText2 and ImageNet-1k | perplexity / CE loss: 29.94 -> 30.22; 11.30 -> 11.37; 6.31 -> 6.94; 10.09 -> 10.85; 0.723 -> 0.728 | Hurt |
| Increase compression using HBFP6 with 2:4 rather than 50% unstructured sparsity | OPT-125M · WikiText2 | perplexity increase: 4.86; 6.93 | Hurt |
| Increase compression using HBFP6 with 2:4 rather than 50% unstructured sparsity | OPT-6.7B · WikiText2 | perplexity increase: 0.79; 6.12 | Hurt |
| Use post-training quantization rather than quantizing during S→Q sparse fine-tuning (ablation) | OPT-125M with 50% sparsity and HBFP8/HBFP6 · WikiText2 validation | perplexity: 33.51 -> 30.46; 36.20 -> 32.51 | Helped |
| Quantize during Q→S fine-tuning rather than using post-training quantization (ablation) | OPT-125M with 50% sparsity and HBFP8/HBFP6 · WikiText2 validation | perplexity: 39.04 -> 37.48; 41.97 -> 40.86 | Helped |
| Apply quantization before magnitude-based sparsity rather than after | Magnitude-based one-shot pruning · WikiText2 | perplexity: OPT-125M 50% HBFP6: 32.51 -> 40.86; OPT-125M 2:4 HBFP6: 34.58 -> 55.64 | Hurt |
| Change operation order for Wanda pruning | Wanda · WikiText2 | perplexity: OPT-125M 50% INT8: 39.29 -> 40.01; LLaMA-2-7B 50% INT8: 6.47 -> 6.46 | No clear effect |
| Change operation order for SparseGPT pruning | SparseGPT · WikiText2 | perplexity: OPT-125M 50% INT8: 33.22 -> 33.54; LLaMA-2-7B 50% INT8: 6.51 -> 6.53 | No clear effect |
| Increase ViT compression to 75% or 1:4 sparsity with HBFP4 | ViT-B/16 · ImageNet-1k | CE loss: 1.094 -> 1.058; 1.163 -> 1.149; 1.365 -> 1.272; 1.725 -> 1.446 | Hurt |
Surprising findings
- Quantization before sparsity (Q→S) can increase perplexity substantially, e.g., OPT-125M with 2:4 and HBFP6 reaches 55.64 versus 34.58 for S→Q.
- Wanda and SparseGPT, which use activation-aware pruning criteria, are largely insensitive to the order of operations.
Limitations
- Orthogonality threshold is a heuristic; the theory gives upper bounds and does not characterize when orthogonality holds.
- Experiments are restricted to 50% and 2:4 sparsity with sparse fine-tuning, plus limited higher-compression tests on ViT and ResNet; heterogeneous schemes are excluded.
- Q→S fine-tuning was unstable, so training iterations and learning rates were limited for fair comparison.
Experimental setup
Datasets: WikiText2 ImageNet-1k
Benchmarks: WikiText2 ImageNet-1k
Metrics: perplexity cross-entropy loss accuracy orthogonality threshold
Baselines: SparseGPT Wanda Dense FP32
Builds on: SparseGPT Wanda HBFP MX formats
Closest papers by meaning
- Network Memory Footprint Compression Through Jointly Learnable Codebooks and Mappings (ICLR 2024 · similarity 0.89)
- Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.89)
- Unified Scaling Laws for Compressed Representations (NeurIPS 2025 · similarity 0.88)
- Quantization without Tears (CVPR 2025 · similarity 0.88)
- Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.88)
- SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.