Research map / Architectures and efficiency / Pruning and efficient tuning

BMRS: Bayesian Model Reduction for Structured Pruning

NeurIPS 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Structured network pruning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BMRS, a threshold-free structured pruning method that applies Bayesian model reduction to multiplicative-noise variational dropout, with two reduced priors (truncated log-normal and truncated log-uniform).

Problem

Structured pruning of neural networks usually requires hand-tuned thresholds to balance accuracy and compression; a principled, threshold-free criterion is needed.

Approach

Combines variational multiplicative-noise pruning with Bayesian model reduction: the change in log-evidence under a reduced prior that collapses noise variables to zero gives a closed-form pruning rule. A truncated log-normal reduced prior (BMRS_N) gives threshold-free pruning, while a truncated log-uniform reduced prior (BMRS_U) with precision bound p1 gives tunable, more aggressive compression.

Key results

• BMRS_N achieves compression of about 76% on CIFAR10 MLP and 63% on CIFAR10 LeNet-5 with accuracy near the unpruned baseline and no threshold tuning (e.g., 43.59% vs 44.94% for the MLP).
• On TinyImageNet with pretrained ResNet-50, BMRS_U-4 reaches 76.12% compression at 50.82% accuracy versus 53.01% unpruned, while SNR reaches 77.85% at 50.54%.
• On CIFAR10 with pretrained ViT, BMRS_U-4 reaches 67.16% compression at 94.83% accuracy versus 94.80% unpruned, while SNR reaches 73.03% at 94.78%.

Measured effects reported in the paper

Surprising findings

• L2 magnitude pruning at the same compression rate as BMRS_N degrades to random-chance accuracy in all settings.
• BMRS_U rankings have very low or negative Spearman correlation with SNR rankings, suggesting it is suited to a single keep/remove split rather than providing a ranking.

Limitations

• Multiplicative noise adds extra variational parameters that increase training time and storage.
• Only applied to linear layers and convolutional filters; more complex structures and hierarchical priors are not explored.
• Baseline pruning criteria (e.g., Hessian- or gradient-based) are limited in comparison.

Experimental setup

Datasets: MNIST Fashion-MNIST CIFAR10 TinyImageNet

Benchmarks: MNIST Fashion-MNIST CIFAR10 TinyImageNet

Metrics: parameter compression (%) test accuracy Spearman rank correlation training and inference runtime (ms)

Baselines: L2 magnitude pruning E[theta] thresholding SNR pruning None (no compression)

Builds on: variational dropout Bayesian model reduction log-uniform variational dropout (Molchanov et al.) SNR-based structured pruning

Closest papers by meaning

• Structured Bayesian Pruning via Log-Normal Multiplicative Noise (NeurIPS 2017 · similarity 0.90)
• Data-Efficient Structured Pruning via Submodular Optimization (NeurIPS 2022 · similarity 0.90)
• Pruning via Sparsity-indexed ODE: a Continuous Sparsity Viewpoint (ICML 2023 · similarity 0.88)
• Probabilistic Neural Pruning via Sparsity Evolutionary Fokker-Planck-Kolmogorov Equation (ICLR 2025 · similarity 0.88)
• Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compression (CVPR 2025 · similarity 0.88)
• Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.87)