Research map / Architectures and efficiency / Neural architecture search
Search for Efficient Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free evolutionary architecture search that selects structural subnets (rows/columns) of pretrained LLMs and reforms their weights with an ADMM-based reconstruction using 128 calibration samples.
Problem
Compressing LLMs by discovering efficient architectures (structural subnets) without retraining, where traditional NAS is too costly for large models and most compression methods focus on weight optimization rather than architecture.
Approach
Initialize subnets by importance scores (WoodFisher/SparseGPT-style saliency), run evolutionary search with a mask mutation operator over inheriting ratios, depth, and channel indices using perplexity on a few samples, then reform inherited weights via ADMM to compensate for omitted weights.
Key results
- On LLaMA-7B at 60% inheriting ratio, WikiText2 perplexity of 10.21 vs 38.27 for LLM-Pruner and 279.52 for SliceGPT (FLAP 12.34).
- At 80% inheriting ratio on LLaMA-7B, average zero-shot accuracy 59.71 vs 58.58 for FLAP and 46.99 for SliceGPT.
- On LLaMA-65B at 50% ratio, WikiText2 perplexity 6.45 at 60% vs FLAP 8.11, showing gains on the largest models tested.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the searched subnet instead of FLAP at an 80% inheriting ratio | LLaMA-7B · WikiText2 | perplexity: 7.40 -> 6.89 | Helped |
| Use the searched subnet instead of FLAP at an 80% inheriting ratio | LLaMA-7B · Common-sense reasoning average | accuracy: 58.58 -> 59.71 | Helped |
| Use the searched subnet instead of FLAP at an 80% inheriting ratio | LLaMA-13B · PTB | perplexity: 23.33 -> 22.66 | Helped |
| Use the searched subnet instead of FLAP at an 80% inheriting ratio | Vicuna-7B · WikiText2 | perplexity: 8.90 -> 8.20 | Helped |
| Use the searched subnet instead of SliceGPT at an 80% inheriting ratio | OPT-1.3B · WikiText2 | perplexity: 20.17 -> 19.23 | Helped |
| Use the searched subnet instead of FLAP at a 60% inheriting ratio | LLaMA-7B · WikiText2 | perplexity: 12.34 -> 10.21 | Helped |
| Use the searched subnet instead of FLAP at a 50% inheriting ratio | LLaMA-30B · WikiText2 | perplexity: 13.41 -> 8.01 | Helped |
| Search on PTB training data rather than WikiText2 (ablation) | LLaMA-7B · WikiText2, 80% inheriting ratio | perplexity: 6.89 -> 7.08 | Hurt |
| Search on PTB training data rather than WikiText2 (ablation) | LLaMA-7B · PTB, 80% inheriting ratio | perplexity: 36.06 -> 34.87 | Helped |
| Increase reformation calibration examples from 128 to 512 or 1024 (ablation) | LLaMA-7B subnet · WikiText2 | perplexity | No clear effect |
| Increase ADMM reformation iterations toward 30 (ablation) | LLaMA-7B subnet, 80% inheriting ratio · WikiText2 | perplexity | Helped |
| Increase ADMM reformation iterations beyond 30 to 50 (ablation) | LLaMA-7B subnet, 80% inheriting ratio · WikiText2 | perplexity | No clear effect |
Surprising findings
- SliceGPT's WikiText2-calibrated results degrade sharply at low inheriting ratios (e.g., 279.52 at 60% on LLaMA-7B), while the proposed method remains usable.
Limitations
- Search cost increases with model size, taking more time for large models.
- Does not search the number of attention heads or embedding/output layers by design.
- Results for SliceGPT are sensitive to calibration dataset.
Experimental setup
Datasets: WikiText2 PTB
Benchmarks: WikiText2 PTB BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenBookQA
Metrics: perplexity zero-shot accuracy average accuracy GPU memory generation speed
Baselines: LLM-Pruner SliceGPT FLAP
Builds on: SparseGPT WoodFisher LLM-Pruner SliceGPT FLAP
Closest papers by meaning
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.91)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.90)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.90)
- Puzzle: Distillation-Based NAS for Inference-Optimized LLMs (ICML 2025 · similarity 0.90)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.90)
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.