Research map / Architectures and efficiency / Pruning and efficient tuning
SliMOO: Interpretable Multi-Objective Evolutionary Search for LLM Depth Pruning
Guanchen Li, Yixing Xu, Xuanwu Yin, Dong Li, Emad Barsoum
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SliMOO, an interpretable multi-objective evolutionary search for LLM depth pruning that accounts for task-dependent layer contributions.
Problem
Depth pruning of LLMs to reduce serving cost, where existing methods ignore that transformer layers contribute differently across tasks and tend to optimize for a single average objective.
Approach
Treats depth pruning as multi-objective search over which layers to remove, using evolutionary search so that task-specific performance trade-offs are preserved rather than collapsed into one objective; the abstract describes the approach as interpretable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-objective Pareto search with TLIP instead of EvoPress | SliMOO · Llama-3.1-70B 25% layer pruning | Math avg: 24.33 -> 34.40 | Improved |
| Use SliMOO instead of EvoPress, language perplexity | SliMOO · Llama-3.1-70B 25% layer pruning | Language perplexity avg: 9.88 -> 9.89 | No clear effect |
| Add TLIP prior guidance on top of NSGA-II (ablation) | SliMOO · Llama-3.1-70B 25% layer pruning | Code avg: 36.02 -> 39.14 | Improved |
| Use NSGA-II instead of scalarized mean objective (ablation) | SliMOO · Llama-3.1-70B 25% layer pruning | Math avg: 29.14 -> 32.67 | Improved |
| Use NSGA-II instead of hypervolume-only selection (ablation) | SliMOO · Llama-3.1-70B 25% layer pruning | Math avg: 30.77 -> 32.67 | Improved |
| Use adaptive temperature instead of fixed tau=1 (ablation) | SliMOO · Llama-3.1-70B 25% layer pruning | Math avg: 32.61 -> 34.40 | Improved |
| Increase generations from 20 to 30 (ablation) | SliMOO · Llama-3.1-70B 25% layer pruning | Math avg: 29.49 -> 34.40 | Improved |
| Increase offspring from 64 to 128 (ablation) | SliMOO · Llama-3.1-70B 25% layer pruning | Knowledge avg: 66.90 -> 65.77 | Mixed |
| Halve calibration budget from 64K to 32K tokens (ablation) | SliMOO · Llama-3.1-70B 25% layer pruning | All domains: 34.40 -> 34.40 | No clear effect |
| Reduce calibration to 8K tokens per domain (ablation) | SliMOO · Llama-3.1-70B 25% layer pruning | Math avg: 34.40 -> 29.04 | Hurt |
| Use math-dominant calibration allocation (ablation) | SliMOO · Llama-3.1-70B 25% layer pruning | Math avg: 34.40 -> 35.16 | Mixed |
| Rerun baselines with mixed multi-domain calibration (EvoPress-mix) (ablation) | EvoPress · Llama-3.1-70B 25% layer pruning | Math avg: 24.33 -> 27.89 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- EvoPress — EvoPress: Accurate Dynamic Model Compression via Evolutionary Search (ICML 2025) · better on 10 of 11 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.90)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.90)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.90)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.90)
- Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing (ICLR 2025 · similarity 0.90)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
