Research map / Architectures and efficiency / Pruning and efficient tuning
NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM
Connor Holmes, Minjia Zhang, Yuxiong He, Bo Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ADMM-based NxMTransformer instead of one-shot magnitude pruning (ASP) for 4:2 sparsity | BERT-base · GLUE dev average | average score: 78.8 -> 80.5 | Improved |
| use NxMTransformer instead of ASP on small tasks (CoLA, STS-B, MRPC, RTE) | BERT-base 4:2 · GLUE small tasks | accuracy: 51.7 -> 55.3 on CoLA | Improved |
| use NxMTransformer instead of ASP on RTE | BERT-base 4:2 · RTE | accuracy: 63.9 -> 68.6 | Improved |
| use NxMTransformer instead of ASP on large tasks (MNLI, QNLI, SST-2) | BERT-base 4:2 · MNLI-m | accuracy: 83.3 -> 82.3 | No clear effect |
| use NxMTransformer instead of ASP on QNLI | BERT-base 4:2 · QNLI | accuracy: 90.6 -> 90.4 | No clear effect |
| use NxMTransformer instead of ASP, median over 7 random seeds on tasks with fewer than 10,000 examples | BERT-base 4:2 · small GLUE tasks | accuracy (median): +1.9 points | Improved |
| use unstructured 50% ADMM sparsity instead of 4:2 semi-structured sparsity | BERT-base · GLUE dev average | average score: 80.5 -> 81.3 | Improved |
| sparsify dense BERT-base to 4:2 with NxMTransformer | BERT-base · GLUE dev average | average score: 81.8 -> 80.5 | Hurt |
| apply NxMTransformer 4:2 sparsity to DistilBERT | DistilBERT · GLUE dev average | average score: 78.5 -> 76.6 | Hurt |
| use 4:2 sparse BERT-base (NxMTransformer) instead of DistilBERT at same parameter count | BERT-base vs DistilBERT · GLUE dev average | average score: 78.5 -> 80.5 | Improved |
| increase mask similarity across ADMM iterations by tuning rho (up to about 99% similarity) (ablation) | NxMTransformer · CoLA and QNLI | Matthews correlation / accuracy | Improved |
| increase mask similarity beyond about 99% (rho too large) (ablation) | NxMTransformer · CoLA and QNLI | accuracy | Hurt |
Closest papers by meaning
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.88)
- SliceGPT: Compress Large Language Models by Deleting Rows and Columns (ICLR 2024 · similarity 0.87)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.87)
- SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models (ICML 2024 · similarity 0.87)
- The Sparsity Whisperer (NeurIPS 2026 · similarity 0.87)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
