Research map / Architectures and efficiency / Pruning and efficient tuning
Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks using the Marginal Likelihood
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpaM, which trains neural networks to maximize a Laplace-approximated marginal likelihood with structured sparsity-inducing priors so they become more sparsifiable, and OPD, a cheap posterior-precision-based pruning criterion.
Problem
Standard training does not make networks sparsifiable, so pruning hurts accuracy, especially at high sparsity; most prior work only designs better pruning criteria.
Approach
Use the Laplace-approximated marginal likelihood as a differentiable objective to learn layer-, unit-, and parameter-wise prior precisions (automatic Occam's razor) during training, and reuse the posterior precision as an Optimal Brain Damage-style pruning score (OPD).
Key results
- At 95% sparsity on CIFAR-10 ResNet, SpaM with OPD reaches 89.98% accuracy versus 37.52% for OPD with MAP training.
- SpaM-OPD at 99% sparsity reaches 66.28% on CIFAR-10 ResNet, versus 17.32% for MAP-OPD.
- Structured SpaM-OPD yields 10x-20x savings in FLOPS and memory with minimal performance loss on LeNet/FashionMNIST.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with SpaM rather than MAP | ResNet with OPD pruning · CIFAR-10, 95% sparsity | accuracy: 37.52 (±2.34) -> 89.98 (±0.61) | Helped |
| train with SpaM rather than MAP | ResNet with OPD pruning · CIFAR-10, 99% sparsity | accuracy: 17.32 (±1.01) -> 66.28 (±5.89) | Helped |
| train with SpaM rather than MAP | ResNet with GraSP pruning · CIFAR-10, 99% sparsity | accuracy: 15.75 (±0.80) -> 41.48 (±7.95) | Helped |
| train with SpaM rather than MAP | ResNet with Magnitude pruning · CIFAR-10, 99% sparsity | accuracy: 16.12 (±0.90) -> 41.78 (±7.20) | Helped |
| train with SpaM rather than MAP | ResNet with SNIP pruning · CIFAR-10, 99% sparsity | accuracy: 12.58 (±0.36) -> 11.90 (±0.51) | Hurt |
| train with SpaM rather than MAP | ResNet with Random pruning · CIFAR-10, 99% sparsity | accuracy: 10.27 (±0.17) -> 9.81 (±0.61) | Hurt |
| use parameter-wise rather than layerwise prior (ablation) | SpaM + OPD on LeNet · MNIST, 99% sparsity | accuracy: 36.25 -> 75.92 | Helped |
| use parameter-wise rather than scalar prior (ablation) | SpaM + OPD on LeNet · MNIST, 99% sparsity | accuracy: 16.12 -> 75.92 | Helped |
| use Diag with a parameter-wise prior rather than KFAC with a layerwise prior (ablation) | SpaM + OPD on LeNet · MNIST, 99% sparsity | accuracy: 28.16 -> 75.92 | Helped |
| use Diag rather than KFAC with parameter-wise prior (ablation) | SpaM + OPD on MLP-Mixer (2 blocks) · MNIST, 99% sparsity | accuracy: 96.97 -> 98.50 | Helped |
| train with SpaM rather than MAP | MLP-Mixer with OPD pruning · MNIST, 99% sparsity | accuracy: 57.66 -> 98.50 | Helped |
| train with SpaM rather than MAP | ResNet with OPD pruning · CIFAR-10, 99% sparsity | NLL: 2.62 ± 0.0106 -> 3.43 ± 0.0179 | Hurt |
Surprising findings
- Random pruning performs no better under SpaM training, consistent with SpaM improving the criterion-dependent sparsifiability rather than random pruning.
Limitations
- Inherits Laplace approximation limits such as capturing only a single local posterior mode.
- Numerical instability of Hessian computations with low-precision weights.
- Additional training cost compared with MAP training.
Experimental setup
Datasets: CIFAR-10 MNIST FashionMNIST IMDB
Benchmarks: CIFAR-10 MNIST FashionMNIST IMDB
Metrics: test accuracy negative log-likelihood expected calibration error Brier score
Baselines: MAP training random pruning magnitude pruning SNIP GraSP SynFlow L1 regularization
Builds on: Laplace approximation for deep learning marginal likelihood model selection Optimal Brain Damage KFAC ARD
Closest papers by meaning
- Effective Sparsification of Neural Networks With Global Sparsity Constraint (CVPR 2021 · similarity 0.90)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.87)
- Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.87)
- Effective Interplay between Sparsity and Quantization: From Theory to Practice (ICLR 2025 · similarity 0.86)
- Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity (ICML 2024 · similarity 0.86)
- BMRS: Bayesian Model Reduction for Structured Pruning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.