Research map / Large language models / Language, safety and interpretability

Compressing Large Language Models by Joint Sparsification and Quantization

ICML 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes JSQ, a joint sparsification and quantization framework for LLMs that uses a salience-with-activation-range sparsity metric and a simulated-annealing-searched activation editor to compress models at high compression ratios.

Problem

Single-technique LLM compression (sparsification or quantization alone) degrades sharply at high compression ratios, and outliers hinder both sparsification and quantization when combined naively.

Approach

Combine unstructured sparsity and quantization in one pipeline, with a Salience with Activation Range (SAR) metric that penalizes weights whose removal enlarges the output activation range, and a search-based activation editor that clamps activation outliers with a per-layer strength found by simulated annealing.

Key results

• At 3.24T MACs (about 7.96x reduction from LLaMA-7B's 14.64T), JSQ reaches WikiText perplexity 11.04 and 62.56 average accuracy, while Wanda and SparseGPT collapse to perplexity in the thousands.
• On LLaMA-33B at 4.77T MACs, JSQ achieves 70.27 average accuracy versus 68.61 for OmniQuant at 22.7G size, with JSQ at 17.0G.
• Ablations on LLaMA-7B show SAR improves WikiText perplexity by 0.71, activation editing by 0.58, and simulated annealing search by 0.25.

Measured effects reported in the paper

Surprising findings

• Under the same computation budget, quantization-only methods outperform pure sparsity methods, and pure sparsity methods such as Wanda and SparseGPT collapse at high compression ratios.
• Directly clamping activation outliers before compression improves perplexity up to a point, and removing too many critical outliers then degrades it.

Limitations

• The editing strength choice set R must be predefined, which requires prior knowledge.
• The compressed model occasionally generates meaningless or repetitive tokens.
• Search requires repeated compression and evaluation on calibration data, adding compression cost.

Experimental setup

Datasets: C4 WikiText2 PIQA BoolQ MMLU HellaSwag ARC-easy ARC-challenge WinoGrande

Benchmarks: WikiText2 PIQA BoolQ MMLU HellaSwag ARC-easy ARC-challenge WinoGrande

Metrics: perplexity zero-shot accuracy MACs model size

Baselines: Wanda LLM-Pruner SparseGPT SmoothQuant OmniQuant Wanda+SmoothQuant SparseGPT+SmoothQuant

Builds on: Wanda SmoothQuant SparseGPT OmniQuant GPTQ

Closest papers by meaning

• Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.92)
• Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.91)
• SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.90)
• Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.90)
• Accuracy is Not All You Need (NeurIPS 2024 · similarity 0.90)
• Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models (NeurIPS 2024 · similarity 0.90)