Research map / Large language models / Language, safety and interpretability
Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ACBench, a benchmark evaluating how post-training quantization (GPTQ, AWQ, SmoothQuant) and pruning (Wanda, SparseGPT, magnitude) affect LLM agentic capabilities across tool use, workflow generation, long-context understanding, and real-world applications.
Problem
Existing LLM compression benchmarks focus on perplexity and NLU tasks, leaving agentic capabilities (workflow, tool use, long-context, real-world application) of compressed LLMs unevaluated.
Approach
Systematically evaluate compressed open LLMs across agentic benchmarks (T-Eval, WorfBench, LongBench, LongGenBench, Needle-in-Haystack, AgentBoard) and add three analysis metrics (ERank, top-k ranking consistency, energy-based logit shift) to explain compression effects.
Key results
- 4-bit quantization preserves workflow generation and tool use with about 1-3% drop but degrades real-world application accuracy by 10-15%, per the abstract.
- Distilled DeepSeek-R1-Distill-Qwen-7B drops average tool-use accuracy from about 70% to 43.6% versus Qwen2.5-7B.
- On AgentBoard PDDL, DeepSeek-R1-Distill-Qwen2.5-7B progress rate drops from 33% to 1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill Qwen2.5-7B into DeepSeek-R1-Distill-Qwen2.5-7B | Qwen2.5-7B · T-Eval | average accuracy: 70% -> 43.6% | Hurt |
| distill Qwen2.5-7B into DeepSeek-R1-Distill-Qwen2.5-7B | Qwen2.5-7B · WorfBench | average F1: 44% -> 20% | Hurt |
| compress Qwen2.5-3B with GPTQ or AWQ | Qwen2.5-3B · GSM8K | score: 61% → 11% | Hurt |
| distill Qwen2.5-7B into DeepSeek-R1-Distill-Qwen2.5-7B | Qwen2.5-7B · PDDL | progress rate: 33% to 1% | Hurt |
| apply AWQ to Qwen2.5-3B | Qwen2.5-3B · AgentBoard | progress rate: 23% | Mixed |
| apply 4-bit quantization | LLMs · Workflow generation and tool use | performance drop: 1\%--3\% drop | No clear effect |
| apply 4-bit quantization | LLMs · Real-world applications | accuracy: 10\%--15\% drop | Hurt |
| use quantization (GPTQ or AWQ) instead of sparsification | LLMs · T-Eval tool use | performance | Helped |
| use magnitude-based pruning | LLMs · WorfBench workflow generation | performance | Hurt |
| use AWQ instead of other compression methods | LLMs · LongGenBench | performance | Helped |
| apply compression | LLMs · Needle-in-the-Haystack | information retrieval performance | Hurt |
| distill Qwen2.5-7B into a DeepSeek-R1 distilled model | Qwen2.5-7B · LongBench single-document QA, multi-document QA, and synthetic tasks | performance | Hurt |
Surprising findings
- Distilled DeepSeek-R1 reasoning models perform markedly worse than their undistilled base models on agentic tasks.
- Smaller DeepSeek-R1-Distill-Qwen-1.5B outperforms its 7B and 8B counterparts on workflow generation.
Limitations
- Only post-training quantization and pruning; no quantization-aware training.
- Default configurations used; group size and other hyperparameters not explored.
- Findings restricted to vLLM-compatible methods, excluding e.g. QuaRot.
Experimental setup
Datasets: Pile validation subset (calibration) T-Eval ToolBench ToolAlpaca WorfBench LongBench LongGenBench Needle-in-the-Haystack AgentBoard
Benchmarks: T-Eval WorfBench LongBench LongGenBench Needle-in-the-Haystack AgentBoard
Metrics: accuracy F1 progress rate success rate ERank top-k Jaccard/ranking consistency energy difference
Baselines: GPTQ AWQ SmoothQuant SparseGPT Wanda Magnitude pruning
Builds on: GPTQ AWQ SparseGPT Wanda AgentBoard
Closest papers by meaning
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.92)
- Fix the Structural Bottleneck: Context Compression via Explicit Information Transmission (NeurIPS 2026 · similarity 0.90)
- When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Models (ICLR 2026 · similarity 0.90)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.90)
- LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment (NeurIPS 2024 · similarity 0.90)
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.