Research map / Architectures and efficiency / Pruning and efficient tuning

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision

ICML 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Structured network pruning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an iterative, data-free low-rank refinement that alternately updates a mask-constrained sparse matrix and adds a rank-k correction to recover performance lost when pruning LLaMA models.

Problem

Pruning large language models with magnitude or other importance scores creates a gap between dense and sparse weights, degrading performance especially at high sparsity, and prior recovery methods need data, distillation, or break sparsity patterns.

Approach

Approximate the dense matrix as a mask-constrained sparse matrix plus a rank-k matrix, and iteratively refine the sparse part using the SVD of the residual with a linearly increasing rank schedule r(t), preserving the binary mask throughout and requiring no calibration data.

Key results

• At 50% unstructured sparsity on LLaMA-7B, the method reduces WikiText-2 perplexity from 17.29 (magnitude) to 7.97 (-53.9%), and at 60% and 70% sparsity it reduces perplexity by 92.0% and 99.6%.
• With Wanda on LLaMA-7B at 50% sparsity, perplexity drops from 7.26 to 6.92 (-4.7%); at 60% it drops 16.1%.
• On Llama-3.1-8B at 50% magnitude sparsity, GSM8K accuracy rises from 1.3% to 6.5%, while the dense model reaches 49.8%.

Measured effects reported in the paper

Surprising findings

• Zero-shot SVD of the residual performs close to the proposed iterative method with Wanda at 50% sparsity, and PCP with mask underperforms the simple SVD baseline.
• Inference speedup is limited: unstructured pruning can be slower than dense computation on GPUs without sparse acceleration.

Limitations

• Adds k(m+n) low-rank parameters per matrix, so the effective sparsity drops slightly (e.g., 4.9% more parameters at k=128); gains at 50% sparsity over Wanda are small.
• Rank k is fixed by hand (k=128) with no automatic per-layer rank selection.
• Evaluation is limited to LLaMA-family models and WikiText-2 perplexity plus a few benchmarks.

Experimental setup

Datasets: allenai/c4 (calibration for Wanda only)

Benchmarks: WikiText-2 TruthfulQA GSM8K ARC-C MMLU HellaSwag WinoGrande ARC-e

Metrics: perplexity accuracy average accuracy

Baselines: Magnitude pruning Wanda SparseGPT Zero-shot SVD PCP with mask

Builds on: Wanda Principal Component Pursuit magnitude pruning SparseGPT

Closest papers by meaning

• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.92)
• DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration (NeurIPS 2025 · similarity 0.92)
• SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.92)
• ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models (NeurIPS 2024 · similarity 0.92)
• A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.91)
• Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.91)