Research map / Large language models / Language, safety and interpretability
Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyWIA, a structured LLM pruning method that adaptively fuses fine-grained (Taylor/Fisher per-weight) and coarse-grained (group-level) importance scores via a training-free attention fusion module, followed by LoRA recovery.
Problem
Structured pruning of LLMs relies on a single granularity of weight importance, causing larger performance drops than unstructured pruning and producing layer sparsity distributions that conflict with fine-grained criteria.
Approach
Combine fine-grained and coarse-grained gradient-based importance scores through an attention-based fusion that computes a per-weight mixing ratio from the two gradient signals, without training the fusion projections, within a dependency-grouping structured pruning pipeline.
Key results
- At 50% pruning of LLaMA-7B, HyWIA reaches 51.80% average zero-shot accuracy vs 48.98% for LLM-Pruner Element^2 and WikiText2 PPL 29.35 vs 45.70.
- At 20% pruning of LLaMA-7B, HyWIA gets 62.44% average accuracy, versus 60.05 for LoRAPrune and 62.08 for FLAP.
- Adaptive fusion beats no-adaptive fusion by about 1.2 points at 50% pruning (51.80 vs 50.42 average) on LLaMA-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hybrid-grained adaptive pruning instead of FLAP | LLaMA-7B · 50% pruning | average accuracy: 50.37 -> 51.80 | Helped |
| Use hybrid-grained adaptive pruning instead of LLM-Pruner Element² | LLaMA-7B · 20% pruning | average accuracy: 59.23 -> 62.44 | Helped |
| Use hybrid-grained adaptive pruning instead of LLM-Pruner Element² | Vicuna-7B · 20% pruning | average accuracy: 57.78 -> 61.35 | Helped |
| Use hybrid-grained adaptive pruning instead of LLM-Pruner Element² | Bloom-7b1 · 25% pruning | average accuracy: 50.19 -> 51.39 | Helped |
| Use hybrid-grained adaptive pruning instead of fixed fusion (ablation) | LLaMA-7B · 20% pruning | average accuracy: 61.20 -> 62.44 | Helped |
| Use hybrid-grained adaptive pruning instead of fixed fusion (ablation) | LLaMA-7B · 50% pruning | average accuracy: 50.42 -> 51.80 | Helped |
| Increase input sample count from 10 to 50 (ablation) | HyWIA · LLaMA-7B, 20% pruning | average accuracy: 60.63 -> 62.44 | Helped |
| Increase pruning ratio from 20% to 50% (ablation) | HyWIA · LLaMA-7B | average accuracy: 62.44 -> 51.80 | Hurt |
Surprising findings
- The attention-fusion module uses randomly initialized, never-trained projection matrices yet is claimed to adapt to inputs.
- Fine-grained and coarse-grained importance estimates allocate parameters across layers in nearly opposite directions.
Limitations
- Fusion projections are untrained, so the claimed adaptivity rests on fixed random mappings of gradients rather than learned behavior.
- Evaluation focuses on zero-shot classification and perplexity; gains over LLM-Pruner at 20% pruning are small in some comparisons (e.g., Bloom at 25% shows higher perplexity than baseline).
Experimental setup
Datasets: yahma/alpaca-cleaned WikiText2 PTB BoolQ PIQA HellaSwag WinoGrande ARC-e ARC-c OBQA
Benchmarks: WikiText2 PTB BoolQ PIQA HellaSwag WinoGrande ARC-e ARC-c OBQA
Metrics: perplexity zero-shot accuracy average accuracy
Baselines: LLM-Pruner (Element^2) LLM-Pruner (Vector) LoRAPrune FLAP Wanda SparseGPT SLEB Compresso Magnitude L2 Block
Builds on: LLM-Pruner LoRA Wanda SparseGPT
Closest papers by meaning
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.94)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.94)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.93)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.93)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.92)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.