Research map / Architectures and efficiency / Pruning and efficient tuning
Soft Weight-Sharing for Neural Network Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| soft weight-sharing with learned mixture prior instead of Han et al. prune-quantize pipeline | LeNet-300-100 · MNIST | compression rate: 40 -> 64 | Helped |
| soft weight-sharing instead of Han et al. | LeNet-5-Caffe · MNIST | compression rate: 39 -> 162 | Helped |
| soft weight-sharing instead of Han et al. | LeNet-300-100 · MNIST | top-1 error after compression (%): 1.58 -> 1.94 | Hurt |
| soft weight-sharing instead of Han et al. | LeNet-5-Caffe · MNIST | top-1 error after compression (%): 0.74 -> 0.97 | Hurt |
| apply compression to light ResNet | WRN-16-4 · CIFAR-10 | top-1 error (%): 6.48 -> 8.50 | Hurt |
| retrain without hyper-priors (mixtures collapse) (ablation) | LeNet-300-100 · MNIST | accuracy: 0.9811 -> 0.9806 | No clear effect |
| Bayesian-optimize 13 hyper-parameters incl. hyper-priors with 40 epochs (ablation) | LeNet-5-Caffe · MNIST | compression vs accuracy loss | Helped |
| alternative initialization distributing means by equal probability mass (ablation) | soft weight-sharing · MNIST | compression | No clear effect |
Closest papers by meaning
- Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters (ICLR 2019 · similarity 0.89)
- Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-Based Approach (CVPR 2020 · similarity 0.89)
- Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (NeurIPS 2022 · similarity 0.88)
- OPQ: Compressing Deep Neural Networks with One-shot Pruning-Quantization (AAAI 2021 · similarity 0.88)
- “Learning-Compression” Algorithms for Neural Net Pruning (CVPR 2018 · similarity 0.88)
- Network Memory Footprint Compression Through Jointly Learnable Codebooks and Mappings (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.