Research map / Architectures and efficiency / Pruning and efficient tuning
EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EvoPress, a (1+λ) evolutionary search with level-switch mutation and multi-step selection that finds non-uniform per-layer compression profiles (layer dropping, unstructured sparsity, quantization) under a global size constraint, with a convergence proof.
Problem
Dynamic, non-uniform LLM compression typically assumes error monotonicity/additivity across layers, which fails in practice; existing methods (heuristic block scores, OWL, DP/SPDY) yield suboptimal allocations at high compression.
Approach
Treat dynamic compression as a black-box optimization over per-unit compression levels and solve it with an elitist (1+λ) evolutionary algorithm using level-switch mutation (preserves size constraint) and aggressive multi-step selection on increasingly large KL-divergence calibration batches, with a provable linear-function convergence bound.</port>
Key results
- At 70% unstructured sparsity on Llama-3-8B, EvoPress reaches Wiki2 PPL 28.76 vs 48.07 for OWL and task average 50.8 vs 48.4.
- At 3-bit quantization on Llama-3-8B, EvoPress reaches Wiki2 PPL 7.49 vs 12.19 for uniform GPTQ and 29.00 for DP, with task average 64.3 vs 60.2 uniform.
- For depth pruning on Mistral-7B-v0.3, EvoPress keeps functional submodels even when removing half the model, where all baselines fail beyond 31.25% sparsity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EvoPress instead of uniform 70% layer sparsity | Mistral-7B-v0.3 · 70% unstructured sparsity | Wiki2 perplexity: 23.08 -> 14.42 | Helped |
| Use EvoPress instead of OWL 70% layer sparsity | Mistral-7B-v0.3 · 70% unstructured sparsity | Task average accuracy: 51.9 -> 53.8 | Helped |
| Use EvoPress instead of OWL 70% layer sparsity | Llama-3-8B · 70% unstructured sparsity | C4 perplexity: 52.32 -> 33.72 | Helped |
| Use EvoPress instead of OWL 70% layer sparsity | Phi-3-Medium-14B · 70% unstructured sparsity | Task average accuracy: 55.4 -> 59.8 | Helped |
| Use EvoPress instead of uniform quantization | Mistral-7B-v0.3 · 3-bit quantization | Wiki2 perplexity: 5.54 -> 5.21 | Helped |
| Use EvoPress instead of uniform quantization | Llama-3-8B · 3-bit quantization | Wiki2 perplexity: 12.19 -> 7.49 | Helped |
| Use EvoPress instead of uniform quantization | Phi-3-Medium-14B · 3-bit quantization | C4 perplexity: 9.05 -> 9.00 | Helped |
| Prune more blocks using the shown configuration | Llama-3-8B · Depth pruning example | Wiki2 perplexity: 188.01 -> 24.39 | Helped |
Surprising findings
- Pruning strictly more blocks can improve perplexity, so error monotonicity (additivity of per-layer errors) does not generally hold for LLM compression.
- Dynamic quantization allocation differs from sparsity allocation: the first block is compressed most under quantization but among the least under unstructured sparsity.
Limitations
- Theory covers only linear fitness functions and the (1+1)/(1+λ) setting with idealized noise-free fitness.
- Depth pruning search keeps equal numbers of attention and MLP removals and only allows swaps between same-type modules.
- Combining different compression approaches in one search space and finer-grained structured pruning are left to future work.
Experimental setup
Datasets: FineWeb-Edu WikiText-2 C4
Benchmarks: WikiText-2 C4 WinoGrande PiQA HellaSwag ARC-easy ARC-challenge
Metrics: perplexity zero-shot accuracy KL divergence
Baselines: Shortened Llama ShortGPT Weight Subcloning Sliding Window Cosine Similarity OWL Uniform DP (SPDY-style) SLEB BlockPruner
Builds on: SPDY OWL SparseGPT GPTQ
Closest papers by meaning
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.90)
- Language Modeling Is Compression (ICLR 2024 · similarity 0.90)
- LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment (NeurIPS 2024 · similarity 0.90)
- LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression (NeurIPS 2026 · similarity 0.90)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.90)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.