Research map / Large language models / Language, safety and interpretability
Pruner-Zero: Evolving Symbolic Pruning Metric From Scratch for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses genetic programming over a symbolic search space of weight, gradient and activation operations to evolve a post-training LLM pruning metric (Pruner-Zero) that beats Wanda and SparseGPT without weight updates.
Problem
Post-training pruning metrics for LLMs are hand-designed through trial and error and are highly sensitive to their exact formulation, so better metrics are hard to find systematically.
Approach
Formulate pruning-metric design as symbolic regression: represent metrics as expression trees over W, G, X with unary/binary operators, search them with genetic programming using WikiText2 perplexity as fitness, and prune redundant equivalent forms with an Opposing Operation Simplification strategy.
Key results
- On LLaMA-7B at 50% unstructured sparsity, Pruner-Zero reaches WikiText2 perplexity 6.95 vs. 7.26 for Wanda and 7.22 for SparseGPT (SparseGPT uses weight updates).
- On LLaMA-2-70B at 50% unstructured sparsity, perplexity is 3.82 vs. 3.98 for both Wanda and SparseGPT, and zero-shot average accuracy exceeds the dense model (71.10 vs. 67.08).
- The searched metric sigma-scaled form ||W|x|W|| x sigma(|G|) is found in under 5 minutes per candidate evaluation on LLaMA-2-7B using two NVIDIA 4090 GPUs for the search.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Pruner-Zero instead of Wanda | LLaMA-2-7B at 50% sparsity · WikiText2 | perplexity: 6.42 -> 6.26 | Helped |
| Use Pruner-Zero instead of SparseGPT | LLaMA-2-70B at 50% sparsity · WikiText2 | perplexity: 3.98 -> 3.82 | Helped |
| Use Pruner-Zero instead of SparseGPT | LLaMA-2-7B at 2:4 sparsity · WikiText2 | perplexity: 10.17 -> 10.52 | Hurt |
| Use Pruner-Zero instead of SparseGPT | LLaMA-2-7B at 50% sparsity · zero-shot mean accuracy | mean accuracy (%): 56.24 -> 58.87 | Helped |
| Use Pruner-Zero instead of SparseGPT | LLaMA-2-70B at 50% sparsity · zero-shot mean accuracy | mean accuracy (%): 67.28 -> 71.10 | Helped |
| Use Pruner-Zero instead of Wanda | LLaMA-2-13B, 8-shot · GSM8K | accuracy: 0.1312 -> 0.1403 | Helped |
| Add Opposing Operation Simplification (OOS) (ablation) | Pruner-Zero on LLaMA-2-7B · WikiText2, 50% unstructured sparsity | perplexity: 6.8395 -> 6.7079 | Helped |
| Add LoRA fine-tuning (ablation) | Pruner-Zero on LLaMA-7B · WikiText2, 50% sparsity | perplexity: 6.95 -> 6.74 | Helped |
| Add LoRA fine-tuning (ablation) | Pruner-Zero on LLaMA-7B · WikiText2, 2:4 sparsity | perplexity: 10.61 -> 8.24 | Helped |
| Add LoRA fine-tuning (ablation) | Pruner-Zero on LLaMA-2-7B · WikiText2, 50% sparsity | perplexity: 6.26 -> 6.41 | Hurt |
| Add LoRA fine-tuning (ablation) | Pruner-Zero on LLaMA-2-7B · WikiText2, 2:4 sparsity | perplexity: 10.52 -> 7.73 | Helped |
| Use evolutionary search instead of random search (ablation) | Pruner-Zero metric search · search convergence | iterations to converge: fewer than 100 iterations vs. approximately 300 iterations | Helped |
Surprising findings
- Metrics that differ only slightly in normalization can produce drastically different perplexities (e.g., 6.94 vs 9377), showing high format sensitivity.
- On LLaMA-2-7B at 2:4 sparsity, SparseGPT (with weight updates) slightly beats Pruner-Zero (10.17 vs 10.52).
Limitations
- Search is driven by perplexity on LLaMA-2-7B with WikiText2, so the discovered metric may be tuned to that setting.
- Requires gradient statistics precomputed on 128 calibration samples, adding preprocessing cost relative to Wanda.
- Gains over Wanda are modest (e.g., 0.3 perplexity at 7B), and SparseGPT remains better on some 2:4 settings.
Experimental setup
Datasets: WikiText2 C4 GSM8K BoolQ RTE HellaSwag WinoGrande ARC OBQA
Benchmarks: WikiText2 BoolQ RTE HellaSwag WinoGrande ARC OBQA GSM8K
Metrics: perplexity zero-shot accuracy in-context learning accuracy
Baselines: Magnitude SparseGPT Wanda GBLM-Pruner
Builds on: Wanda SparseGPT GBLM-Pruner AutoML-Zero EMQ
Closest papers by meaning
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.93)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.92)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.92)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.92)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.91)
- Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.