Research map / Architectures and efficiency / Pruning and efficient tuning
Learning Semi-Structured Sparsity for LLMs via Shared and Context-Aware Hypernetwork
Lu Sun, Jun Sakuma
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add layer and component context-aware embeddings to the shared hypernetwork (ablation) | HyperPrune · Wikitext-2 (LLaMA-2 7B, 2:4) | perplexity: 11.36 -> 10.11 | Improved |
| add feature outlier regularization (ablation) | HyperPrune · Wikitext-2 (LLaMA-2 7B, 2:4) | perplexity: 11.06 -> 10.11 | Improved |
| add continual pruning regularization (ablation) | HyperPrune · Wikitext-2 (LLaMA-2 7B, 2:4) | perplexity | Improved |
| add layer embeddings vs component embeddings (ablation) | HyperPrune · Wikitext-2 (LLaMA-2 7B, 2:4) | perplexity | Mixed |
| replace SparseGPT with HyperPrune | LLaMA-2 7B 2:4 pruning · Zero-shot mean of 7 tasks | mean accuracy: 51.00 -> 53.76 | Improved |
| replace MaskPro with HyperPrune | LLaMA-2 7B 2:4 pruning · Zero-shot mean of 7 tasks | mean accuracy: 52.81 -> 53.76 | Improved |
| replace MaskPro with HyperPrune | LLaMA-2 7B 2:4 pruning · Wikitext-2 | perplexity: 12.29 -> 10.11 | Improved |
| replace MaskPro with HyperPrune | LLaMA-2 13B 2:4 pruning · Zero-shot mean of 7 tasks | mean accuracy: 58.97 -> 59.25 | Improved |
| replace Pruner-Zero with HyperPrune | LLaMA-2 13B 2:4 pruning · Wikitext-2 | perplexity: 7.53 -> 7.60 | Hurt |
| replace Pruner-Zero with HyperPrune | LLaMA-2 70B 2:4 pruning · Wikitext-2 | perplexity: 4.87 -> 5.13 | Hurt |
| replace Pruner-Zero with HyperPrune | LLaMA-2 70B 2:4 pruning · Zero-shot mean of 7 tasks | mean accuracy: 67.69 -> 68.57 | Improved |
| prune LLaMA-2-7B with HyperPrune 2:4 sparsity instead of dense | LLaMA-2 7B · End-to-end latency on A100 | latency (ms): 248 ms -> 174 ms (1.43x) | Improved |
Closest papers by meaning
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.94)
- Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity (ICML 2024 · similarity 0.92)
- MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models (NeurIPS 2024 · similarity 0.92)
- Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.92)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.92)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
