Research map / Architectures and efficiency / Pruning and efficient tuning
Sparse maximal update parameterization: A holistic approach to sparse training dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes sparse maximal update parameterization (SµPar), which rescales weight init and learning rates with width and density so activations, gradients, and updates stay scale-stable, enabling hyperparameter transfer from small dense proxies to sparse models.
Problem
Sparse neural networks suffer vanishing signal propagation and their optimal hyperparameters shift with sparsity level, so reusing dense hyperparameters is costly and suboptimal, making sparse training at scale hard to tune.
Approach
Extends µP to random unstructured static sparsity by scaling hidden-weight initialization variance by 1/(width multiplier × density multiplier) and Adam learning rate by 1/(width × density multiplier), so forward activations, backward gradients, and weight updates are Θ(1) across width and sparsity, allowing µTransfer of dense-tuned HPs to sparse models.
Key results
- SµPar reaches 11.9% relative validation loss improvement over SP and 1.9% over µP at 99.2% sparsity for a 610M-parameter LLM.
- SµPar's optimal learning rate stays near 2^-6 across sparsity levels from 0 to 98.4%, whereas SP and µP optimal learning rates drift with sparsity.
- At 99% sparsity, SµPar keeps about 40% average downstream accuracy versus 34% for SP (random baseline 30%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SµPar initialization and learning-rate corrections instead of SP | GPT-like transformer language model · SlimPajama validation, 99.2% random unstructured sparsity | validation loss: 11.9% relative loss improvement | Helped |
| Apply SµPar initialization and learning-rate corrections instead of µP | GPT-like transformer language model · SlimPajama validation, 99.2% random unstructured sparsity | validation loss: 1.9% better loss | Helped |
| Apply SµPar instead of SP | GPT-like transformer language model · Average downstream tasks, 99.2188% sparsity | accuracy: 0.34 -> 0.40 | Helped |
| Apply SµPar instead of µP | GPT-like transformer language model · Average downstream tasks, 99.2188% sparsity | accuracy: 0.39 -> 0.40 | Helped |
| Apply SµPar instead of µP | GPT-like transformer language model · Average downstream tasks, 87.5% sparsity | accuracy: 0.44 -> 0.44 | No clear effect |
| Use only SµPar initialization correction with µP, omitting the SµPar learning-rate correction (ablation) | µP · Sparse training across sparsity levels | optimal hyperparameter transfer | Hurt |
| Use only SµPar learning-rate correction with µP, omitting the SµPar initialization correction (ablation) | µP · Sparse training across sparsity levels | optimal hyperparameter transfer | Hurt |
| Apply SµPar to dynamic sparse training methods RigL and GMP | RigL and GMP · Dynamic sparsity across sparsity levels | optimal learning-rate transfer | Hurt |
Surprising findings
- Dense µP hyperparameters, when adjusted via SµPar, are optimal for all sparse models tested, so dense-tuned HPs transfer directly.
- For dynamic sparse training (RigL, GMP), none of SP, µP, or SµPar transfers optimal learning rate across sparsity; SµPar overcorrects with higher sparsity.
Limitations
- SµPar assumes Gaussian-like random unstructured masks; it overcorrects for dynamic sparse methods (RigL, GMP).
- Derivation relies on law-of-large-numbers arguments that weaken when non-zero weights per neuron are small.
- Scaling factors must be derived per optimizer; only sparsified linear projection layers are considered.
Experimental setup
Datasets: SlimPajama
Benchmarks: ARC-easy LAMBADA RACE PIQA BoolQ
Metrics: validation loss train loss downstream accuracy
Baselines: Standard parameterization (SP) µP RigL GMP Iso-Parameter scaling
Builds on: maximal update parameterization (µP) µTransfer Feature Learning Desiderata Evci et al. sparse initialization
Closest papers by meaning
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.91)
- Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together! (ICLR 2023 · similarity 0.90)
- A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.88)
- Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.88)
- Navigating Extremes: Dynamic Sparsity in Large Output Spaces (NeurIPS 2024 · similarity 0.88)
- Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.