Research map / Architectures and efficiency / CNN design
Variational Dropout Sparsifies Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Sparse VD instead of the original dense model | LeNet-300-100 · MNIST | Error %: 1.64 -> 1.92 | Mixed |
| Use Sparse VD instead of pruning | LeNet-300-100 · MNIST | Parameter reduction ratio: 12 -> 68 | Helped |
| Use Sparse VD instead of DNS | LeNet-300-100 · MNIST | Parameter reduction ratio: 56 -> 68 | Helped |
| Use Sparse VD instead of the original dense model | LeNet-5-Caffe · MNIST | Error %: 0.80 -> 0.75 | Helped |
| Use Sparse VD instead of SWS | LeNet-5-Caffe · MNIST | Parameter reduction ratio: 200 -> 280 | Helped |
| Use Sparse VD instead of a dense VGG-like model | VGG-like, k=1 · CIFAR-10 | Error %: 7.3% -> 7.3% | No clear effect |
| Replace multiplicative-noise parameterization with Additive Noise Reparameterization (ablation) | Sparse VD · Two-layer, 1000-neuron fully connected network | Variational lower bound | Helped |
| Initialize from a pretrained network rather than random initialization (ablation) | Sparse VD · VGG-like networks | Accuracy | Helped |
| Threshold weights with log alpha=3 to zero at test time (ablation) | Sparse VD · Neural network experiments | Performance | No clear effect |
| Train with Sparse VD rather than Binary Dropout | Fully connected MNIST and VGG-like CIFAR-10 networks · Random labels | Ability to fit random labels | Hurt |
Closest papers by meaning
- Structured Dropout Variational Inference for Bayesian Neural Networks (NeurIPS 2021 · similarity 0.92)
- Variational Bayesian dropout: pitfalls and fixes (ICML 2018 · similarity 0.90)
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (ICML 2016 · similarity 0.90)
- Variational Bayesian Dropout With a Hierarchical Prior (CVPR 2019 · similarity 0.90)
- Dropout Inference in Bayesian Neural Networks with Alpha-divergences (ICML 2017 · similarity 0.90)
- Stochastic Modified Equations and Dynamics of Dropout Algorithm (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.