Research map / Architectures and efficiency / RNNs and normalization
Most Activation Functions Can Win the Lottery Without Excessive Depth
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the L+1 construction instead of the 2L construction | LeNet lottery-ticket pruning · MNIST | accuracy: ReLU: 97.78 ± 0.05 vs 97.96 ± 0.02; LReLU: 97.63 ± 0.08 vs 97.84 ± 0.06; tanh: 98.07 ± 0.07 vs 98.14 ± 0.03; sigmoid: 98 | No clear effect |
| Use the L+1 construction instead of the 2L construction | LeNet lottery-ticket pruning · MNIST | parameters: ReLU: 1106.5 ± 0.9 vs 119.2 ± 0.04; LReLU: 1102.4 ± 0.9 vs 119.2 ± 0.1; tanh: 660.3 ± 0.4 vs 67.0 ± 0.06; sigmoid: 669.7 | Hurt |
| Use the L+1 construction instead of the 2L construction | ResNet18 classifier lottery-ticket pruning · tiny-ImageNet test | accuracy: ReLU: 73.09 ± 0.04 vs 73.06 ± 0.05; LReLU: 72.96 ± 0.04 vs 72.92 ± 0.03; tanh: 73.75 ± 0.04 vs 73.72 ± 0.08; sigmoid: 72 | No clear effect |
| Use the L+1 construction instead of the 2L construction | ResNet18 classifier lottery-ticket pruning · tiny-ImageNet test | parameters: ReLU: 19.14 ± 0.005 vs 21.8 ± 0.004; LReLU: 19.14 ± 0.01 vs 21.8 ± 0.006; tanh: 11.36 ± 0.01 vs 10.98 ± 0.01; sigmoid: 1 | Mixed |
| Use the L+1 construction to exploit a deeper, sparser target representation | Circle-task lottery-ticket pruning · circle test | parameters: 98393 ± 288 vs 18891 ± 133 | Helped |
| Use the L+1 construction to exploit a deeper, sparser target representation | Circle-task lottery-ticket pruning · circle test | accuracy: 99.5 ± 0.6 vs 99.8 ± 0.01 | Helped |
Closest papers by meaning
- Convolutional and Residual Networks Provably Contain Lottery Tickets (ICML 2022 · similarity 0.93)
- Optimal Lottery Tickets via Subset Sum: Logarithmic Over-Parameterization is Sufficient (NeurIPS 2020 · similarity 0.92)
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (ICLR 2019 · similarity 0.92)
- On the Existence of Universal Lottery Tickets (ICLR 2022 · similarity 0.92)
- Proving the Lottery Ticket Hypothesis for Convolutional Neural Networks (ICLR 2022 · similarity 0.90)
- Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Sparse Neural Networks (NeurIPS 2021 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.