Research map / Architectures and efficiency / Pruning and efficient tuning
Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that permuting a lottery-ticket mask (via activation matching plus REPAIR) to align with a new random initialization's basin improves sparse training from that new initialization on CIFAR-10/100 and ImageNet.
Problem
Lottery ticket masks found on one random initialization do not transfer to a different random initialization, so sparse training from a new random init performs poorly.
Approach
Hypothesizes the failure is basin misalignment; permutes the LTH/IMP mask with the permutation that aligns two dense trained models (activation matching, with REPAIR for variance collapse) before sparse training from a new initialization.
Key results
- On ResNet20x1 CIFAR-10 at 97% sparsity, permuted mask reaches about 86.5% vs 85.6% naive, while LTH reaches 89.3% (rewind k=25).
- ResNet20x8 CIFAR-100 at 97% sparsity: permuted 76.8% vs naive 78.7% at k=100? (permuted ~76.8% vs naive ~72.8% at k=100), with LTH at 78.5%.
- On ResNet50 ImageNet at 80% sparsity, permuted top-1 is 69.87% vs naive 69.13% at k=10; permuted ensembles show higher functional diversity than LTH at 90% sparsity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Permute the LTH mask to align it with a new random initialization | Sparse ResNet20 training · CIFAR-10 (90% sparsity, k=75) | accuracy: 88.81 ± 0.17 -> 90.03 ± 0.07 | Helped |
| Permute the LTH mask to align it with a new random initialization | Sparse ResNet20 training · CIFAR-100 (90% sparsity, k=50) | accuracy: 59.19 ± 0.41 -> 61.53 ± 0.65 | Helped |
| Permute the LTH mask to align it with a new random initialization | Sparse VGG11 training · CIFAR-10 (90% sparsity, k=50) | accuracy: 89.80 ± 0.05 -> 91.01 ± 0.06 | Helped |
| Permute the LTH mask to align it with a new random initialization | Sparse ResNet50 training · ImageNet (90% sparsity, k=25) | Top-1 accuracy: 64.77 -> 66.37 | Helped |
| Use an ensemble of permuted-mask models rather than one LTH model | ResNet20×1 · CIFAR-10 (90% sparsity) | ensemble accuracy: 91.43 -> 91.75 | Helped |
| Use an ensemble of permuted-mask models rather than one LTH model | ResNet20×4 · CIFAR-100 (90% sparsity) | ensemble accuracy: 75.99 -> 77.85 | Helped |
Surprising findings
- Once variance collapse is corrected with REPAIR, the dense solution and LTH solution remain linearly connected, contrary to prior conclusions that did not account for variance collapse.
- Permuted masks give more functionally diverse ensembles than the LTH mask, reaching ensemble accuracy comparable to IMP solutions.
Limitations
- Finding the permutation requires training two dense models, which is computationally expensive.
- Activation matching is greedy/approximate, leaving a gap to LTH on ImageNet.
- Unstructured sparsity is not supported well by current hardware; VGG11 naive baseline unstable at high sparsity.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet
Benchmarks: CIFAR-10 CIFAR-100 ImageNet
Metrics: test accuracy top-5 accuracy loss barrier disagreement KL divergence JS divergence ensemble accuracy
Baselines: LTH (lottery ticket mask and init) naive (non-permuted LTH mask on new init) IMP
Builds on: Lottery Ticket Hypothesis Iterative Magnitude Pruning Git Re-Basin REPAIR Linear Mode Connectivity
Closest papers by meaning
- Dual Lottery Ticket Hypothesis (ICLR 2022 · similarity 0.89)
- Calibrating the Rigged Lottery: Making All Tickets Reliable (ICLR 2023 · similarity 0.89)
- Why Random Pruning Is All We Need to Start Sparse (ICML 2023 · similarity 0.88)
- Find A Winning Sign: Sign Is All We Need to Win the Lottery (ICLR 2025 · similarity 0.88)
- No Free Prune: Information-Theoretic Barriers to Pruning at Initialization (ICML 2024 · similarity 0.87)
- On the Sparsity of the Strong Lottery Ticket Hypothesis (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.