Research map / Architectures and efficiency / Pruning and efficient tuning

BaWA: Automatic Optimizing Pruning Metric for Large Language Models with Balanced Weight and Activation

ICML 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Layer-wise pruning of LLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BaWA, a one-shot unstructured/N:M pruning metric that balances weight and activation via magnitude normalization and outlier regularization, with zeroth-order search of three power factors per layer.

Problem

One-shot post-training pruning of billion-scale LLMs is limited by pruning metrics that symbolically combine weight magnitude and activation norms, which ignore imbalanced weight channel magnitudes and the outsized influence of activation outliers.

Approach

Replace the simple product |W|·||X||2 with a metric that normalizes weight magnitudes by input- and output-channel L2 norms and raises norms to learnable power factors (outlier regularization), with the power factors tuned per block by a zeroth-order optimizer minimizing block output error.

Key results

• On Mistral-7B with 2:4 sparsity, BaWA reduces WikiText-2 perplexity by 2.49 versus Wanda and improves average zero-shot accuracy by 3.08%.
• At 70% unstructured sparsity on LLaMA-7B, BaWA reaches perplexity 57.84 versus 74.79 for Wanda.
• BaWA's power factor search completes in about 16 minutes for LLaMA2-70B on a single GPU.

Measured effects reported in the paper

Surprising findings

• BaWA is almost insensitive to the number of calibration samples; even a single sample gives satisfactory results.
• Fixed power factor of 0.1 degrades perplexity by 24.1% relative to the 0.5 baseline, showing high sensitivity to power factor choice.

Limitations

• Power factor search adds minutes of calibration cost compared with purely training-free Wanda.
• In some zero-shot cases SparseGPT with weight reconstruction still beats BaWA, so the metric alone does not dominate all settings.

Experimental setup

Datasets: C4

Benchmarks: WikiText-2 HellaSwag PIQA ARC BoolQ RTE Winogrande MMLU

Metrics: perplexity zero-shot accuracy mean accuracy

Baselines: Magnitude SparseGPT Wanda GBLM-Pruner RIA Pruner-Zero DSnoT ADMM-Iter

Builds on: Wanda SparseGPT OmniQuant MeZO

Closest papers by meaning

• AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.91)
• PAT: Pruning-Aware Tuning for Large Language Models (AAAI 2025 · similarity 0.91)
• A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.91)
• Pruner-Zero: Evolving Symbolic Pruning Metric From Scratch for Large Language Models (ICML 2024 · similarity 0.91)
• Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.91)
• An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.90)