Research map / Architectures and efficiency / Pruning and efficient tuning
Masks, Signs, And Learning Rate Rewinding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Disentangles mask learning from parameter optimization in iterative pruning, showing Learning Rate Rewinding (LRR) flips parameter signs early, is robust to sign perturbations, and proves in a single-neuron toy model that LRR escapes bad sign configurations that defeat IMP.
Problem
Iterative pruning methods (IMP, weight rewinding, LRR) couple mask identification and parameter optimization, so it is unclear why LRR outperforms IMP and whether its advantage lies in mask finding or parameter training.
Approach
LRR continues training from the previous cycle's weights with a reset learning rate, which lets it flip parameter signs early when the network is overparameterized; this enables more reliable optimization even on random masks, and a closed-form single-neuron analysis shows LRR succeeds in more sign configurations than IMP.
Key results
- LRR outperforms IMP/WR across CIFAR10, CIFAR100, and Tiny ImageNet; LRR-rand (rewinding BN) outperforms IMP-rand on random masks.
- In the single hidden neuron toy model, IMP succeeds only when a(0)>0 and w1(0)>0, while LRR succeeds whenever a(0)>0.
- LRR flips parameter signs primarily in early pruning iterations and signs stabilize early, whereas IMP shows unstable back-and-forth sign flipping.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continue training from the previous pruning iteration instead of rewinding weights (LRR vs. WR/IMP) | ResNet18/ResNet50 · CIFAR10, CIFAR100, Tiny ImageNet | accuracy | Helped |
| Use LRR's learned mask instead of IMP's mask, while optimizing parameters with IMP | IMP · CIFAR10 | accuracy | Helped |
| Use LRR's learned mask instead of IMP's mask, while optimizing parameters with IMP | IMP · CIFAR100 below 95% sparsity | accuracy | Helped |
| Rewind batch-normalization parameters after each pruning iteration (ablation) | LRR with the IMP mask · CIFAR10, CIFAR100 | accuracy | Helped |
| Train the IMP mask with LRR rather than IMP | IMP mask · Tiny ImageNet | accuracy | Helped |
| Train a random mask by continuing parameters from prior pruning iterations instead of starting from initial values | LRR-rand vs. IMP-rand · CIFAR10, CIFAR100 | accuracy | Helped |
| Randomly perturb parameter signs at low sparsity (ablation) | LRR and IMP · CIFAR10 | accuracy | No clear effect |
| Randomly perturb parameter signs at low sparsity (ablation) | LRR and IMP · CIFAR100 | accuracy | Hurt |
| Randomly perturb parameter signs at 83% sparsity (ablation) | LRR vs. IMP · CIFAR10 | accuracy | Helped |
| Randomly perturb parameter signs at 83% sparsity (ablation) | LRR vs. IMP · CIFAR100 | accuracy | Mixed |
| Rewind only weight magnitudes while retaining LRR's learned signs and mask (ablation) | IMP · CIFAR10, CIFAR100 | accuracy | No clear effect |
| Train a one-dimensional ReLU neuron from a random balanced initialization rather than using a favorable positive-positive sign configuration | Single hidden neuron · Noisy target φ(x) | learning success: 3/4 | Hurt |
Surprising findings
- Rewinding only weight magnitudes while keeping LRR's learned signs and mask matches LRR performance, suggesting signs carry most of the useful information.
Limitations
- Theory restricted to a single hidden neuron with gradient flow, zero noise and balanced initialization
- Original LRR schedule suffers from diminishing BN weights on larger tasks, requiring proposed modifications (BN rewinding, warmup)
- On CIFAR100 at high sparsity LRR suffers more from sign perturbations than IMP
Experimental setup
Datasets: CIFAR10 CIFAR100 Tiny ImageNet ImageNet
Benchmarks: CIFAR10 CIFAR100 Tiny ImageNet ImageNet
Metrics: test accuracy sparsity fraction of sign flips confidence intervals over 3 runs
Baselines: IMP WR LRR IMP-rand strong lottery tickets
Builds on: Iterative Magnitude Pruning Weight Rewinding Learning Rate Rewinding Lottery Ticket Hypothesis
Closest papers by meaning
- Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask? (ICLR 2023 · similarity 0.86)
- Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM) (NeurIPS 2021 · similarity 0.86)
- Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry (ICML 2025 · similarity 0.85)
- Mask in the Mirror: Implicit Sparsification (ICLR 2025 · similarity 0.85)
- Exact risk curves of signSGD in High-Dimensions: quantifying preconditioning and noise-compression effects (ICML 2025 · similarity 0.84)
- Backdoor Mitigation via Invertible Pruning Masks (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.