Research map / Large language models / Language, safety and interpretability

Evaluating Quantized Large Language Models

ICML 2024 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Comprehensive evaluation of post-training quantization (weight-only, weight-activation, KV cache) across 11 LLM families (125M–180B) on basic NLP, emergent abilities, trustworthiness, dialogue, and long-context tasks, yielding bit-width recommendations.

Problem

Existing evaluations of quantized LLMs cover only a few tensor types (mostly weights/activations) and a narrow set of tasks, leaving unclear how PTQ affects KV cache, trustworthiness, dialogue, and long-context performance across model families and sizes.

Approach

Systematically benchmark weight-only, weight-activation, and KV cache quantization (with uniform group-wise/per-token formats) across many model families and sizes and many task types, complemented by tensor statistics (AbsMax, std, kurtosis) to explain sensitivity, and summarize bit-width recommendations per task.

Key results

• The larger the model, the higher tolerance to weight-only and KV cache quantization, while activation quantization tolerance decreases with size; W4, W4A8, and KV4 are near-lossless (<2% loss) for most tasks.
• W4A4 causes most LLMs to lose essentially all performance, with SmoothQuant recovering some dialogue ability at W4A4 for LLaMA2-13B and 70B.
• For long-context tasks (>=4K), KV8 is near-lossless for some families while KV cache quantization is more harmful than weight-only quantization for most LLMs.

Measured effects reported in the paper

Surprising findings

• Multiple-choice accuracy can increase after quantization (e.g., Mistral-7B on RACE rises 3.5% under W8A8), which the authors attribute to limits of accuracy as a metric.
• Mixtral-8x7B matches LLaMA2-70B in FP16 performance but is more sensitive to weight-only and KV cache quantization.

Limitations

• Only post-training quantization was studied; quantization-aware training was not considered.
• Hyperparameters such as group size were not ablated, and some small models (1B-3B) were not fully covered.
• Bit-width recommendations may not generalize to new LLMs or tasks.

Experimental setup

Datasets: CHID Winogrande RACE LAMBADA PIQA SIQA MMLU CEval GSM8K StrategyQA Hellaswag ARC

Benchmarks: CHID Winogrande RACE LAMBADA PIQA SIQA MMLU CEval GSM8K StrategyQA Hellaswag ARC

Metrics: accuracy GPT-4 score normalized performance Spearman correlation kurtosis

Baselines: AWQ SmoothQuant RTN

Builds on: AWQ SmoothQuant GPTQ LLM.int8() KIVI

Closest papers by meaning

• Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.95)
• QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.92)
• CBQ: Cross-Block Quantization for Large Language Models (ICLR 2025 · similarity 0.91)
• What Makes Quantization for Large Language Model Hard? An Empirical Study from the Lens of Perturbation (AAAI 2024 · similarity 0.91)
• ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals (ICML 2025 · similarity 0.91)
• Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization (NeurIPS 2025 · similarity 0.91)