Research map / Large language models / Language, safety and interpretability
LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents LLMCBench, a six-track benchmark comparing seven LLM compression methods (LLM-Pruner, Wanda, SparseGPT, GPTQ, SmoothQuant, AWQ, OmniQuant) on performance, generalization, training and inference cost, hardware acceleration, and trustworthiness.
Problem
Existing LLM compression papers evaluate on different models, datasets, and metrics, and rarely measure practical efficiency (training cost, real inference memory, library speedups) or trustworthiness, so it is unclear which compression method to use in a given scenario.
Approach
A multi-track benchmark that unifies evaluation protocols and defines overall quadratic-mean metrics per track (performance, generalization, training consumption, inference consumption, hardware acceleration, trustworthiness) to compare sparsity and quantization methods on the same models and datasets.
Key results
- Quantization methods reach OM_perf above 95 (e.g., OmniQuant 99.42, AWQ 99.39, GPTQ 98.58) while the best sparsity method SparseGPT 2:4 reaches 74.73 and unstructured SparseGPT 88.25.
- GPTQ and AWQ achieve generalization scores above 95 (93.80 and 96.13), while SmoothQuant scores 0.82 on the generalization track.
- Wanda needs the least training resources (about 43 overall score), while OmniQuant requires over 300 A800 GPU minutes for compression.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use quantization rather than sparsification | LLM compression · LLMCBench Track 1 | OM_perf: over 95; smaller than 90 | Helped |
| Use sparsification rather than quantization | LLM compression when inference ability is prioritized · LLMCBench Track 1 | OM_ia | Helped |
| Use quantization rather than sparsification | LLM compression when knowledge ability is prioritized · LLMCBench Track 1 | OM_ka | Helped |
| Use weight-only quantization | LLM compression across model families · LLMCBench Track 2 | OM_gen: 93.80; 96.13 | Helped |
| Quantize activations with SmoothQuant | LLM compression across model families · LLMCBench Track 2 | OM_gen: 0.82 | Hurt |
| Apply compression to ChatGLM2-6B | LLM compression methods · ChatGLM2-6B | perplexity | Hurt |
| Choose Wanda for compression | LLM compression training · LLMCBench Track 3 | training consumption: around 43 | Helped |
| Use compression methods that do not require learning | LLM compression training · LLMCBench Track 3 | compression time: more than 300 A800 GPU minutes | Helped |
| Choose SmoothQuant or AWQ | LLM compression with limited training GPU memory · LLMCBench Track 3 | GPU memory usage | Helped |
| Use quantization rather than sparsification | LLM inference · LLMCBench Track 4 | OM_inf: 245.91; 245.11; 246.34; 134.76 | Helped |
| Use structured sparsity rather than unstructured or 2:4 sparsity | LLM inference without special implementation · LLMCBench Track 4 | OM_inf: 161.86; 134.76 | Helped |
| Deploy INT4 quantization | LLM compression · TensorRT-LLM, vLLM, and MLC-LLM | hardware acceleration | Helped |
Surprising findings
- SmoothQuant W8A8 collapses to a perplexity of 1887.53 on LLaMA2-7B in the generalization track, yielding an OM_gen of 0.82.
- LLM-Pruner has the weakest performance track score yet achieves the best trustworthiness score among sparsity methods.
Limitations
- Only seven compression methods (sparsity and PTQ) are covered; KV cache compression, distillation, and QAT are excluded.
- Evaluation uses limited tasks (no coding or math benchmarks), as the authors acknowledge.
- Some sparsity and quantization settings are fixed (e.g., 50% sparsity, W8A16/W4A16 variants), so results depend on those chosen bit-widths and ratios.
Experimental setup
Datasets: WikiText2 C4
Benchmarks: MMLU ARC-challenge ARC-easy HellaSwag PIQA WinoGrande QNLI MNLI WikiText2 AdvGLUE TruthfulQA
Metrics: accuracy perplexity overall metric (quadratic mean of relative scores) GPU memory model size MACs tokens per second training time
Baselines: LLM-Pruner Wanda SparseGPT GPTQ SmoothQuant AWQ OmniQuant
Builds on: SmoothQuant GPTQ AWQ OmniQuant SparseGPT Wanda LLM-Pruner
Closest papers by meaning
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.92)
- EvoPress: Accurate Dynamic Model Compression via Evolutionary Search (ICML 2025 · similarity 0.90)
- Accuracy is Not All You Need (NeurIPS 2024 · similarity 0.90)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.90)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.90)
- Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.