Research map / Architectures and efficiency / Pruning and efficient tuning
DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DenoiseRotator, which trains learnable orthogonal rotations of LLM weights to minimize entropy of normalized importance scores, concentrating importance so that standard pruning (Magnitude, Wanda, SparseGPT) degrades less at 50% unstructured and 2:4 sparsity.
Problem
Post-training pruning of LLMs causes large performance degradation, especially under semi-structured 2:4 sparsity, because existing methods only select which weights to prune and do not change the distribution of parameter importance.
Approach
Learn orthogonal transformations of weight matrices (computationally invariant in Transformers) by minimizing the information entropy of normalized OBD-style importance scores, so importance concentrates on fewer weights before pruning; the rotation is decoupled and plugged into any existing pruning method.
Key results
- On LLaMA-3-70B with SparseGPT at 2:4 sparsity, DenoiseRotator reduces WikiText-2 perplexity from 10.97 to 6.25 (dense 2.86), per the main table.
- At 50% unstructured sparsity on LLaMA3-8B with SparseGPT, zero-shot accuracy rises from 66.88% to 69.58% and perplexity from 9.57 to 7.60.
- Under 2:4 sparsity, Magnitude pruning on LLaMA3-8B improves from 141.96 to 75.23 perplexity when combined with DenoiseRotator, and Wanda on Qwen2.5-72B reaches 6.16 perplexity versus 6.69 without rotation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DenoiseRotator to SparseGPT | LLaMA3-8B pruning at 50% unstructured sparsity · WikiText-2 validation | perplexity: 9.567 -> 7.597 | Helped |
| Add DenoiseRotator to SparseGPT | LLaMA3-8B pruning at 50% unstructured sparsity · Average zero-shot tasks | accuracy: 66.88 -> 69.58 | Helped |
| Add DenoiseRotator to SparseGPT | LLaMA3-8B pruning at 2:4 sparsity · WikiText-2 validation | perplexity: 17.669 -> 10.008 | Helped |
| Add DenoiseRotator to SparseGPT | LLaMA3-8B pruning at 2:4 sparsity · Average zero-shot tasks | accuracy: 55.95 -> 66.11 | Helped |
| Add DenoiseRotator to SparseGPT | LLaMA3-70B pruning · WikiText-2 validation at 50% unstructured sparsity | perplexity: 5.99 -> 4.61 | Helped |
| Add DenoiseRotator to SparseGPT | LLaMA3-70B pruning · WikiText-2 validation at 2:4 sparsity | perplexity: 10.97 -> 6.25 | Helped |
| Train orthogonal rotations for 100 steps instead of 0 (ablation) | DenoiseRotator with SparseGPT on LLaMA3-8B at 50% unstructured sparsity · WikiText-2 validation | perplexity: 9.567 -> 7.701 | Helped |
| Train orthogonal rotations for 100 steps instead of 0 (ablation) | DenoiseRotator with SparseGPT on LLaMA3-8B at 50% unstructured sparsity · Zero-shot tasks | accuracy: 66.88 -> 70.54 | Helped |
| Increase rotation optimization from 100 to 2000 steps (ablation) | DenoiseRotator with SparseGPT on LLaMA3-8B at 50% unstructured sparsity · Zero-shot tasks | accuracy: 70.54 -> 69.58 | Mixed |
| Replace dense rotations with block diagonal rotations using 2 blocks (ablation) | DenoiseRotator with SparseGPT on LLaMA3-8B at 50% unstructured sparsity · WikiText-2 validation | perplexity: 7.597 -> 8.024 | Hurt |
| Replace dense rotations with block diagonal rotations using 4 blocks (ablation) | DenoiseRotator with SparseGPT on LLaMA3-8B at 50% unstructured sparsity · WikiText-2 validation | perplexity: 7.597 -> 8.544 | Hurt |
| Replace dense rotations with block diagonal rotations using 8 blocks (ablation) | DenoiseRotator with SparseGPT on LLaMA3-8B at 50% unstructured sparsity · WikiText-2 validation | perplexity: 7.597 -> 8.882 | Hurt |
Surprising findings
- Magnitude pruning at 2:4 on LLaMA3-8B is worse with DenoiseRotator than Wanda does without it in some cells, and DenoiseRotator sometimes degrades zero-shot accuracy (e.g., LLaMA3-8B 2:4 Magnitude+DenoiseRotator 41.10 vs Magnitude 48.04 average accuracy), which the authors do not fully explain.
- The authors suggest orthogonal matrices act like a random permutation that helps weights align with the 2:4 pattern, though they did not explicitly optimize for 2:4.
Limitations
- Semi-structured sparsity constraints are not explicitly integrated into the rotation optimization.
- Adds about 6.7% parameter overhead per model (extra hidden-size orthogonal matrices per layer) and training cost scales with model size (about 28 hours for LLaMA-3-70B on one A100).
- Experiments are limited to post-training pruning without recovery finetuning in the main results.
Experimental setup
Datasets: C4 WikiText-2
Benchmarks: WikiText-2 perplexity PIQA WinoGrande HellaSwag ARC-e ARC-c
Metrics: perplexity zero-shot accuracy
Baselines: Magnitude Wanda SparseGPT RotPruner
Builds on: SliceGPT QuaRot SpinQuant Wanda SparseGPT Optimal Brain Damage
Closest papers by meaning
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.92)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.92)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.91)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.90)
- PAT: Pruning-Aware Tuning for Large Language Models (AAAI 2025 · similarity 0.90)
- SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.