Research map / Large language models / Language, safety and interpretability
The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM
Kwanhee Lee, Hyeondo Jang, Dongyeop Lee, Dan Alistarh, Namhoon Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace layer-wise surrogate pruners with surrogate-free ADMM (ELSA) | SparseGPT/ALPS · LLaMA-2-7B, 90% sparsity | C4 perplexity: 180.9 (ALPS) -> 23.14 | Improved |
| Use ELSA instead of Wanda + full retraining | Wanda+Full · LLaMA-2-7B, 99% sparsity | C4 perplexity: 71.64 -> 40.10 | Improved |
| Use ELSA instead of Wanda + LoRA | Wanda+LoRA · LLaMA-2-7B, 95% sparsity | WikiText perplexity: 371.0 -> 38.91 | Improved |
| Add objective-aware projection (ablation) | ELSA · LLaMA-3.2-3B, 90% sparsity | perplexity: 65.41 -> 48.69 | Improved |
| Add objective-aware projection (ablation) | ELSA · LLaMA-3.2-3B, 70% sparsity | perplexity: 29.44 -> 28.24 | Improved |
| Use ELSA's uniform allocation instead of EvoPress non-uniform mask (ablation) | ELSA (EvoPress mask) · LLaMA-3-8B, 70% sparsity | C4 perplexity: 29.33 -> 29.09 | Improved |
| Use ELSA instead of non-uniform allocation methods | OWL / EvoPress · LLaMA-3-8B, 70% sparsity | C4 perplexity: 33.72 (EvoPress) -> 29.09 | Improved |
| Use ELSA under N:M semi-structured sparsity (2:4) | ALPS / L-ADMM · LLaMA-2-7B 2:4 | zero-shot avg accuracy: 53.85 (ALPS) vs 49.59 | Hurt |
| Use next-token-prediction objective instead of reconstruction error with more data (ablation) | reconstruction-error minimization · OPT-125M, 90% sparsity | perplexity | Mixed |
| Raise sparsity from 50% to 95% with MACKO kernel | dense LLaMA-2-7B · LLaMA-2-7B on RTX 3090 | throughput: 54.47 -> 216.98 tokens/s | Improved |
Closest papers by meaning
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.92)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.92)
- MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models (NeurIPS 2024 · similarity 0.92)
- SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models (ICML 2024 · similarity 0.92)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.91)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
