Research map / Architectures and efficiency / Pruning and efficient tuning

The Sparsity Whisperer

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Layer-wise pruning of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes difference-informed one-shot LLM pruning (Wisp, Wisp+, Whisper) that scores weights in MLP up/gate projections by pairwise input-difference statistics, with a regularized difference Hessian for second-order reconstruction.

Problem

One-shot LLM weight pruning criteria preserve large activations or layer reconstruction, but ignore the input-difference (separation) computation performed by sparsity-sensitive Wasserstein neurons in MLP up and gate projections.

Approach

Score weights by their contribution to output differences between calibration input pairs (first-order: |W_ij| times norm of input-difference channel; second-order: Hessian built from input differences, lightly regularized with the standard Hessian), applied only to MLP up and gate projections.

Key results

• Whisper improves over SparseGPT in perplexity at every tested sparsity and model size on Llama 2 (50% to 75%), e.g., Llama 2 7B 2:4 PPL 8.18 to 7.56.
• Wisp+ (no weight update) closes 68% of the Wanda-SparseGPT perplexity gap at 65% sparsity on Llama 2 7B, 78% on 13B, 67% on 70B.
• ALPS-Whisper improves ALPS in 8 of 9 settings on Llama 2 (e.g., 70B 2:4 PPL 4.50 to 4.41); RIA-Wisp improves RIA in all nine settings.

Surprising findings

• Difference-informed criteria hurt the MLP down projection and give mixed results on attention, so they are restricted to up and gate projections.
• Nearest-neighbor pairing for Wisp improves perplexity at 2:4 and 65% sparsity, but adds 30-65% pruning time.

Limitations

• Only one-shot pruning on dense models without post-pruning finetuning was studied.
• Difference-informed criteria applied only to MLP up and gate projections; down and attention projections not improved.
• Wisp+ and nearest-neighbor variants add pruning overhead relative to Wanda; MoE models not evaluated.

Experimental setup

Datasets: WikiText-2

Benchmarks: WikiText-2 perplexity ARC Challenge MathQA HellaSwag MMLU TruthfulQA-MC WinoGrande

Metrics: perplexity mean accuracy pruning wall-clock time

Baselines: Magnitude Wanda SparseGPT RIA ALPS

Builds on: Wanda SparseGPT RIA ALPS Wasserstein neurons

Closest papers by meaning

• Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.92)
• Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.92)
• Learn To be Efficient: Build Structured Sparsity in Large Language Models (NeurIPS 2024 · similarity 0.91)
• Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.91)
• R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.90)
• Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.90)