Research map / Architectures and efficiency / Pruning and efficient tuning
SLTrain: a sparse plus low rank approach for parameter and memory efficient pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SLTrain, which parameterizes each linear weight as a low-rank product plus a fixed random-support sparse matrix trained on its nonzero values, for memory- and parameter-efficient LLM pretraining.
Problem
Pretraining LLMs from scratch is memory and parameter intensive, and low-rank parameterizations that save resources usually underperform full-rank training because they restrict weights to a low-dimensional subspace.
Approach
Parameterize weights as W = BA + S, where S has a uniformly random fixed sparsity support and only its non-zero values (with indices) are learned; this keeps the weight high-rank while storing only low-rank factors and sparse values, avoiding dense masks and full-rank matrices.
Key results
- On LLaMA 1B, SLTrain reaches 16.14 PPL with 646M params versus 15.56 PPL for full-rank at 1339M params, and GaLore at 15.64.
- SLTrain with delta=0.1 matches full-rank perplexity on LLaMA 350M and 1B (18.72 vs 18.80; 15.59 vs 15.56) while cutting parameters by about 42-45%.
- With 8-bit SLTrain and per-layer updates, actual memory drops 51%, 58%, 73% for 350M, 1B, 7B models versus full-rank, and 8-bit SLTrain on 7B reaches 27.59 PPL vs 26.87 for 8-bit GaLore with 46G vs 62G memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a sparse factor to low-rank pretraining | SLTrain · C4, LLaMA 60M | perplexity: 78.18 -> 34.15 | Helped |
| Add a sparse factor to low-rank pretraining | SLTrain · C4, LLaMA 130M | perplexity: 45.51 -> 26.04 | Helped |
| Add a sparse factor to low-rank pretraining | SLTrain · C4, LLaMA 350M | perplexity: 37.41 -> 19.42 | Helped |
| Add a sparse factor to low-rank pretraining | SLTrain · C4, LLaMA 1B | perplexity: 142.5 -> 16.14 | Helped |
| Use SLTrain instead of GaLore | LLaMA 7B pretraining · C4, LLaMA 7B (1.4B tokens) | perplexity: 26.87 -> 27.59 | Hurt |
| Add top-3% residual entries by pruning to the low-rank approximation (ablation) | LLaMA 60M weight approximation · C4 validation, LLaMA 60M | perplexity: 36633.04 -> 5293.93 | Helped |
| Add random-3% residual entries by pruning to the low-rank approximation (ablation) | LLaMA 60M weight approximation · C4 validation, LLaMA 60M | perplexity: 36633.04 -> 29121.38 | Helped |
| Train sparse values on fixed top support (ablation) | LLaMA 60M low-rank approximation L0 · C4 validation, LLaMA 60M | perplexity: 36633.04 -> 53.75 | Helped |
| Train sparse values on fixed random support (ablation) | LLaMA 60M low-rank approximation L0 · C4 validation, LLaMA 60M | perplexity: 36633.04 -> 51.98 | Helped |
| Vary the random support used for the sparse factor (ablation) | SLTrain · LLaMA 60M and 130M pretraining | perplexity | No clear effect |
| Increase sparsity ratio from 0.03 to 0.1 (ablation) | SLTrain · C4, LLaMA 350M | perplexity: 19.42 -> 18.72 | Helped |
| Increase sparsity ratio from 0.03 to 0.1 (ablation) | SLTrain · C4, LLaMA 1B | perplexity: 16.14 -> 15.59 | Helped |
Surprising findings
- Fixing a random sparse support and training only its values performs as well as top-support selection (51.98 vs 53.75 PPL), suggesting random support suffices.
- Low-rank plus random sparse pruning of a pretrained model performs poorly, yet training the sparse values from a fixed low-rank base recovers performance.
Limitations
- Throughput is slightly lower than full-rank and GaLore due to sparse entry scatter/gather operations.
- Theoretical guarantees for training with combined low-rank and sparse factors are not established.
- Baseline perplexities for most sizes are taken from prior work rather than re-run under identical settings.
Experimental setup
Datasets: C4 GLUE
Benchmarks: C4 validation perplexity GLUE
Metrics: perplexity parameter count estimated memory actual GPU memory throughput accuracy
Baselines: Full-Rank Low-Rank ReLoRA GaLore 8-bit Adam LoRA
Builds on: LoRA ReLoRA GaLore Scatterbrain LoSparse
Closest papers by meaning
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.90)
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.90)
- Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.89)
- Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (NeurIPS 2025 · similarity 0.89)
- Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining (NeurIPS 2025 · similarity 0.89)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.