Research map / Architectures and efficiency / Pruning and efficient tuning
A Signal Propagation Perspective for Pruning Neural Networks at Initialization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Enforce approximate isometry on the sparse network after LDI initialization and connection-sensitivity pruning (ablation) | LDI-pruned networks · CIFAR-10 and Tiny-ImageNet, five tested architectures | generalization error (%): 8.11 -> 7.99; 11.55 -> 11.47; 10.08 -> 9.85; 8.88 -> 8.78; 44.20 -> 44.12 | Helped |
| Use layerwise dynamical isometry initialization instead of variance-scaling initialization | Connection-sensitivity pruning · CIFAR-10 and Tiny-ImageNet, tested architectures | generalization error (%): 8.16 -> 8.11; 11.89 -> 11.55; 10.43 -> 10.08; 9.13 -> 8.88; 45.08 -> 44.20 | Helped |
| Use scaled Gaussian initialization with γ=10⁰ instead of γ=10⁻¹ (ablation) | Connection-sensitivity pruning on a 7-layer tanh MLP · MNIST | error: 2.61 -> 90.2 | Hurt |
| Replace ground-truth targets with a uniform target distribution (ablation) | Connection-sensitivity pruning at orthogonal initialization · MNIST, MLP K=3, K=5, and K=7 | generalization error: 2.46 -> 2.77; 2.43 -> 2.77; 2.61 -> 2.94 | Mixed |
| Use averaged softmax predictions as the target instead of ground-truth labels (ablation) | Connection-sensitivity pruning at orthogonal initialization · MNIST, MLP K=3, K=5, and K=7 | generalization error: 2.46 -> 3.11; 2.43 -> 3.37; 2.61 -> 3.56 | Hurt |
| Use averaged raw predictions as the target instead of ground-truth labels (ablation) | Connection-sensitivity pruning at orthogonal initialization · MNIST, MLP K=3, K=5, and K=7 | generalization error: 2.46 -> 3.31; 2.43 -> 3.38; 2.61 -> 3.60 | Hurt |
| Use uniform-target unsupervised pruning rather than supervised pruning (ablation) | Orthogonally initialized pruning · CIFAR-10, VGG16, ResNet32, ResNet56, and ResNet110 | generalization error (%): 8.11 -> 8.25; 11.55 -> 11.69; 10.08 -> 11.01; 8.88 -> 8.82 | Mixed |
| Use uniform loss rather than input-dependent unsupervised losses (ablation) | Unsupervised connection-sensitivity pruning · MNIST, 7-layer MLP | generalization error: 3.00 -> 2.94 | Helped |
| Prune a wider architecture to the same parameter budget as dense ResNet20 | CIFAR-10 architecture sculpting · CIFAR-10, equal 269k parameter budget | generalization error: 8.046 -> 7.618 | Helped |
| Use a deeper sculpted architecture rather than the wider Equivalent 1 (ablation) | CIFAR-10 architecture sculpting · CIFAR-10, equal 269k parameter budget | generalization error: 7.618 -> 7.912 | Hurt |
| Prune by connection sensitivity rather than randomly | Layerwise-isometric initialization · Sparse 7-layer MLPs across sparsity levels | mean Jacobian singular value | Helped |
| Enforce approximate isometry on randomly pruned sparse networks (ablation) | Layerwise-isometric random-pruned networks · Sparse 7-layer MLPs | training performance | Helped |
Closest papers by meaning
- Pruning Neural Networks at Initialization: Why Are We Missing the Mark? (ICLR 2021 · similarity 0.92)
- Pruning from Scratch (AAAI 2020 · similarity 0.91)
- SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY (ICLR 2019 · similarity 0.91)
- Robust Pruning at Initialization (ICLR 2021 · similarity 0.91)
- Pruning neural networks without any data by iteratively conserving synaptic flow (NeurIPS 2020 · similarity 0.91)
- How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.