Research map / Architectures and efficiency / Pruning and efficient tuning

SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs

ICLR 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SLoPe, a sparse pretraining method combining static 2:4 (N:M) weight sparsity with a double-pruned backward pass and lazy low-rank adapters added in the final 1% of pretraining, with optimized CUDA kernels for speedup and memory savings.

Problem

Sparse LLM pretraining with N:M structured sparsity loses accuracy, and accelerating both forward and backward passes is hard because row-wise N:M structure is lost on transpose; prior fixes (dense fine-tuning, transposable masks) negate inference savings or add large overheads.

Approach

Prune the transposed already-N:M-sparse weight matrix again (double-pruned backward pass) so both passes use 2:4 sparse kernels, keep the mask static, and add low-rank adapters only in the last 1% of pretraining to recover accuracy without dense fine-tuning or extra inference cost.

Key results

• SLoPe achieves up to 1.25x training speedup and 1.54x inference speedup on OPT models (OPT-66B and OPT-13B), with memory reductions to 0.63x (training) and 0.61x (inference) for OPT-66B and OPT-30B.
• On BERT-Large-Uncased, SLoPe with rank-64 adapters reaches 89.5 SQuAD F1 and 78.2 GLUE, versus 90.44 and 80.22 for dense and 77.4 GLUE with no adapter.
• On GPT2-Small zero-shot tasks, SLoPe with 2.1% adapter rank outperforms Extended SR-STE on most tasks (e.g., PiQA 57.6 vs 55.2).

Measured effects reported in the paper

Surprising findings

• Dynamic-mask Extended SR-STE wastes training compute on weights later pruned, so a static mask with SLoPe reaches lower perplexity for the same budget.
• Low-rank adapters added in only the final 1% of pretraining are sufficient to recover much of the accuracy lost to sparsity.

Limitations

• Accuracy experiments are limited to models up to 774M parameters; large-scale accuracy is not evaluated.
• Accuracy experiments emulate N:M sparsity with custom kernels rather than using hardware sparse kernels, and the double-pruned backward pass is lossy.
• Gains on inference depend on adapter rank, and the paper notes low arithmetic intensity makes small low-rank multiplications inefficient.

Experimental setup

Datasets: OpenWebText WikiCorpus SQuAD v1.1 GLUE MMLU ARC Challenge OpenBookQA

Benchmarks: SQuAD v1.1 GLUE MMLU ARC Challenge OpenBookQA WinoGrande HellaSwag MathQA PiQA RACE

Metrics: end-to-end speedup memory reduction validation perplexity zero-shot accuracy F1 average GLUE score

Baselines: Dense FST Extended SR-STE Wanda Bi-Mask

Builds on: SR-STE FST (Fully Sparse Training) Wanda LoRA FlashAttention-2

Closest papers by meaning

• Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.89)
• R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.89)
• Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples (NeurIPS 2025 · similarity 0.89)
• Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (NeurIPS 2025 · similarity 0.89)
• S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity (NeurIPS 2024 · similarity 0.88)
• SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity (ICML 2025 · similarity 0.88)