Research map / Architectures and efficiency / Pruning and efficient tuning
PermLLM: Learnable Channel Permutation for N:M Sparse Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PermLLM, a post-training N:M sparsity framework that learns channel permutations end-to-end via Sinkhorn-relaxed soft permutations and block-wise parameterization, plugged into Wanda and RIA.
Problem
Handcrafted importance-based channel permutation metrics for N:M sparse LLMs do not reliably reduce the actual output error caused by pruning, and full permutation search is combinatorially expensive.
Approach
Learn channel permutations directly by minimizing dense-vs-sparse output discrepancy (cosine loss), relaxing permutations with Sinkhorn normalization, hardening with Hungarian assignment and STE, and restricting permutations to block-diagonal blocks (size 64) to cut parameters and cost.
Key results
- On 2:4 sparsity, PermLLM_Wanda reduces LLaMA-3.1 8B Wikitext2 perplexity from 23.42 (Wanda) to 14.03 and Qwen-2.5 7B from 24.44 to 13.58.
- PermLLM_Wanda reaches 52.17 average zero-shot accuracy on Qwen-2.5 7B versus 51.25 for SparseGPT and 47.91 for Wanda.
- The custom CUDA kernel for channel permutation gives an 84x speedup over PyTorch, with about 1.67x overall linear-layer acceleration reported for 2:4 LLaMA-2 7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add learnable channel permutation to Wanda pruning | Wanda, 2:4 sparsity · Wikitext2 across the seven evaluated models | perplexity: 16.29 -> 14.27; 11.59 -> 9.41; 9.60 -> 8.06; 12.16 -> 9.39; 9.05 -> 8.20; 23.42 -> 14.03; 24.44 -> 13.58 | Helped |
| Add learnable channel permutation to Wanda pruning | Wanda, 2:4 sparsity · Average zero-shot accuracy across OPT 6.7B, LLaMA 7B, LLaMA-2 7B, LLaMA-3.1 8B, and Qwen-2.5 7B | Average accuracy: 40.09 -> 41.81; 41.37 -> 45.67; 42.12 -> 46.59; 38.91 -> 43.33; 47.91 -> 52.17 | Helped |
| Replace conventional channel permutation with learned permutation | Wanda pruning, 2:4 sparsity · Average zero-shot accuracy across the five evaluated models | Average accuracy: 40.62 -> 41.81; 43.94 -> 45.67; 43.44 -> 46.59; 40.72 -> 43.33; 49.38 -> 52.17 | Helped |
| Use 5 rather than 0 Sinkhorn iterations (ablation) | PermLLM_Wanda · Qwen-2.5 7B, average zero-shot tasks, 2:4 | Average accuracy: 42.96 -> 43.33 | Helped |
| Use 5 rather than 0 Sinkhorn iterations (ablation) | PermLLM_Wanda · LLaMA-3.1 8B, average zero-shot tasks, 2:4 | Average accuracy: 49.18 -> 52.17 | Helped |
| Use 5 rather than 0 Sinkhorn iterations (ablation) | PermLLM_Wanda · Qwen-2.5 7B, Wikitext2, 2:4 | perplexity: 14.12 -> 14.03 | Helped |
| Use block size 128 rather than 64 (ablation) | PermLLM_Wanda · LLaMA-2 7B, average zero-shot tasks, 2:4 | Average accuracy: 46.59 -> 47.09 | Helped |
| Use block size 128 rather than 64 (ablation) | PermLLM_Wanda · LLaMA-2 7B pruning | runtime: 2.5h -> 6h | Hurt |
| Use block size 32 rather than 64 (ablation) | PermLLM_Wanda · LLaMA-2 7B, average zero-shot tasks, 2:4 | Average accuracy: 46.59 -> 43.58 | Hurt |
| Apply learnable permutation only to the last six decoder layers (ablation) | PermLLM_RIA · LLaMA-2 7B, average zero-shot tasks, 2:4 | Average accuracy: 44.30 -> 43.78 | Hurt |
| Apply learnable permutation only to the last six decoder layers (ablation) | PermLLM_RIA · LLaMA-2 7B, Wikitext2, 2:4 | perplexity: 9.60 -> 10.10 | Hurt |
Surprising findings
- Wanda and RIA with channel permutation still underperform SparseGPT on LLaMA-3.1 and Qwen-2.5, which PermLLM reverses.
Limitations
- Training still requires more compute than traditional heuristic channel permutation methods.
- Evaluated mainly on 2:4 and 4:8 sparsity with a limited set of zero-shot tasks.
- Applicability to quantization and other tasks is left unexplored.
Experimental setup
Datasets: C4 Pile Wikitext2
Benchmarks: Wikitext2 HellaSwag ARC-Easy ARC-Challenge OpenBookQA RTE
Metrics: perplexity zero-shot accuracy speedup
Baselines: SparseGPT Wanda RIA Wanda+CP RIA+CP
Builds on: Wanda RIA SparseGPT Sinkhorn normalization straight-through estimator
Closest papers by meaning
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.89)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.88)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.88)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.88)
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.88)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.