Research map / Large language models / Language, safety and interpretability

Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression

ICML 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Evaluates how compression (GPTQ, AWQ quantization; Magnitude, SparseGPT, Wanda pruning) of LLaMA2 13B, LLaMA2 13B Chat, and Vicuna 13B affects eight DecodingTrust trustworthiness dimensions plus MMLU, finding quantization preserves trustworthiness better than pruning.

Problem

Compressed LLMs are evaluated almost only on benign utility; their safety and trustworthiness under compression (stereotype, toxicity, privacy, fairness, ethics, adversarial/OOD robustness, adversarial demonstrations) is largely unknown.

Approach

A systematic trustworthiness evaluation of training-free, data-free-style compression (GPTQ/AWQ quantization and Magnitude/SparseGPT/Wanda 2:4 pruning) across three 13B models using the DecodingTrust benchmark, with refusal-aware metric adjustments and comparisons to 7B models trained from scratch.

Key results

• 4-bit quantization (GPTQ/AWQ) of LLaMA2 13B Chat keeps all trustworthiness dimensions within a 5-point drop, while 50% 2:4 pruning significantly degrades trustworthiness.
• 4-bit GPTQ raises the Ethics score of LLaMA2 13B Chat from 54.1 to 76.3, and quantization improves fairness in few-shot settings by over 0.2 EOD reduction at 16 shots.
• At 3-bit, GPTQ degrades OOD robustness by about 30 points and Toxicity by about 50 points, while AWQ 3-bit keeps most dimensions within 5 points but drops in Adv Demo and Fairness; MT-Bench drops from 6.55 (GPTQ 4-bit) to 2.89 (3-bit).

Measured effects reported in the paper

Surprising findings

• Moderate 4-bit quantization can improve ethics and fairness beyond the dense 13B model.
• 3-bit GPTQ's catastrophic trust losses stem from loss of instruction-following ability, and the risk is not visible in MMLU scores.

Limitations

• Evaluation limited to three 13B models and a small set of compression methods, with only N:M sparsity pruning considered.
• Modified DecodingTrust metrics for fairness and ethics (refusal handling) deviate from the original benchmark.
• Large variance across calibration-set seeds, especially for 3-bit and SparseGPT.

Experimental setup

Datasets: C4 MMLU DecodingTrust

Benchmarks: MMLU DecodingTrust AdvGLUE++ Adversarial Demonstrations OOD robustness Fairness Machine Ethics Privacy Stereotype Toxicity MT-Bench

Metrics: accuracy false positive rate equalized odds difference refusal rate normalized trust score (0-100 points) MT-Bench score

Baselines: Magnitude pruning SparseGPT Wanda LLaMA2 7B dense 13B models

Builds on: DecodingTrust GPTQ AWQ SparseGPT Wanda

Closest papers by meaning

• Position: TrustLLM: Trustworthiness in Large Language Models (ICML 2024 · similarity 0.90)
• Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression (ICML 2025 · similarity 0.89)
• Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.89)
• Accuracy is Not All You Need (NeurIPS 2024 · similarity 0.88)
• Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.88)
• MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models (NeurIPS 2024 · similarity 0.88)