Research map / Large language models / Language, safety and interpretability
Fluctuation-Based Adaptive Structured Pruning for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FLAP, a retraining-free structured pruning framework for LLaMA models that scores input channels by a fluctuation metric, globally standardizes scores for adaptive structure search, and adds baseline-mean bias compensation.
Problem
Retraining-free pruning of LLMs has focused on unstructured sparsity, which needs special hardware; existing structured methods such as LLM-Pruner rely on LoRA fine-tuning, and metrics from unstructured pruning transfer poorly to rows/columns.
Approach
Score each input channel by the sample variance of its input features weighted by the squared norm of the corresponding weight column (fluctuation metric), standardize scores per layer and module for a global adaptive threshold search, and compensate removed channels with a bias equal to the weight times the baseline (mean) activation.
Key results
- At 20% pruning, FLAP on LLaMA-7B reaches WikiText2 perplexity 14.62 vs 19.77 for LLM-Pruner and 17.37 for LLM-Pruner with LoRA, without retraining.
- At 50% pruning on LLaMA-7B, FLAP reaches perplexity 31.80 vs 112.44 for LLM-Pruner (no LoRA) and 38.12 with LoRA; zero-shot average 50.37 vs 48.98 for LLM-Pruner with LoRA.
- At 20% pruning on LLaMA-7B, FLAP's zero-shot average is 62.08 vs 63.25 dense; throughput improves by 31% at 20% and 66% at 50% pruning on A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FLAP instead of Wanda-sp at 50% pruning | LLaMA-7B · WikiText2 validation | perplexity: 366.43 -> 31.80 | Helped |
| Use FLAP instead of LLM-Pruner with LoRA | LLaMA-7B at 50% pruning · WikiText2 validation | perplexity: 38.12 -> 31.80 | Helped |
| Use FLAP instead of LLM-Pruner with LoRA | LLaMA-7B at 50% pruning · Zero-shot average | accuracy: 48.98 -> 50.37 | Helped |
| Use FLAP instead of Wanda-sp | LLaMA-7B at 20% pruning · Zero-shot average | accuracy: 63.35 -> 62.08 | Hurt |
| Use FLAP instead of LLM-Pruner with LoRA | LLaMA-13B at 20% pruning · Zero-shot average | accuracy: 64.02 -> 64.52 | Helped |
| Use adaptive layers and modules (AL-AM) instead of uniform layers and modules (UL-UM) (ablation) | WIFV structured pruning · WikiText2 validation, LLaMA-7B at 50% pruning | perplexity: 57.57 -> 31.80 | Helped |
| Use WIFV instead of WIFN with AL-AM (ablation) | FLAP structured pruning · WikiText2 validation, LLaMA-7B at 50% pruning | perplexity: 34.09 -> 31.80 | Helped |
| Use WIFV instead of IFV with AL-AM (ablation) | FLAP structured pruning · WikiText2 validation, LLaMA-7B at 50% pruning | perplexity: 33.33 -> 31.80 | Helped |
| Add baseline bias compensation (ablation) | FLAP · WikiText2 validation, LLaMA-7B at varying pruning ratios | perplexity | Helped |
| Increase calibration sample count (ablation) | FLAP on LLaMA-7B at 50% pruning · WikiText2 validation | perplexity | Helped |
| Use WikiText2 rather than C4 calibration data (ablation) | FLAP · Zero-shot average | accuracy: 50.37 -> 48.33 | Hurt |
| Use C4 rather than WikiText2 calibration data (ablation) | FLAP · PIQA accuracy | accuracy: 63.82 -> 67.52 | Helped |
Surprising findings
- Wanda-sp at 20% pruning slightly exceeds the unpruned LLaMA-7B zero-shot average (63.35 vs 63.25).
- Bias compensation alone is argued to largely replace LoRA-style retraining, with its benefit growing at higher pruning ratios.
Limitations
- Evaluated mainly on LLaMA family (and one Vicuna model); generality to other architectures is claimed but not shown.
- SparseGPT could not be adapted to structured pruning with reasonable results.
- Zero-shot accuracy varies about ±1% depending on calibration dataset choice.
Experimental setup
Datasets: WikiText2 C4
Benchmarks: WikiText2 BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenbookQA
Metrics: perplexity zero-shot accuracy parameter count memory tokens/s
Baselines: Wanda-sp LLM-Pruner LLM-Pruner (w/ LoRA)
Builds on: Wanda SparseGPT LLM-Pruner Optimal Brain Damage
Closest papers by meaning
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.93)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.93)
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.92)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.92)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.92)
- Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.