Research map / Architectures and efficiency / Pruning and efficient tuning

Find A Winning Sign: Sign Is All We Need to Win the Lottery

ICLR 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Lottery ticket discovery · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AWS, a variant of learning rate rewinding that interpolates normalization-layer parameters with their initialization during training, so that the resulting signed mask lets any randomly initialized network match dense performance on CIFAR-100, Tiny-ImageNet and ImageNet.

Problem

Finding a winning lottery ticket: a sparse subnetwork that, when trained from a random initialization, matches dense generalization. Existing iterative pruning methods either fail at larger scale or apply masks to trained weights rather than initialization.

Approach

Parameter sign information from a sparse mask conveys generalization potential to any random initialization, provided normalization-layer parameters are not randomized; AWS randomly interpolates normalization parameters between trained and initial values during training so the learned signed mask transfers to fresh random weights with low linear-mode-connectivity barriers.

Key results

• On ImageNet with ResNet50 at 0.27 remaining params, a randomly initialized network with the AWS signed mask reaches 74.65% top-1 versus 75.26% for the dense network and 73.20% for the LRR signed mask.
• On ImageNet MLP-Mixer at 0.11 remaining params, AWS signed mask training reaches 58.62% top-1 versus 59.57% dense and 56.91% for the LRR signed mask.
• The AWS-driven signed mask gives comparable performance to the dense network up to about 0.9 sparsity on CIFAR-100 with ResNet-32, whereas GM falls short of dense performance at all sparsity levels.

Measured effects reported in the paper

Surprising findings

• Preserving parameter signs alone does not help when all parameters, including normalization layers, are randomly initialized; signs only transfer useful information when normalization parameters are kept or handled.
• For some architectures (e.g., VGG11-bn), the LRR signed mask on a random initialization performs similarly to the LRR solution, unlike ResNets and MobileNets.

Limitations

• Transfer of the AWS signed mask to transfer learning settings is left for future work.
• Results are reported for a limited set of architectures and datasets (CIFAR-100, Tiny-ImageNet, ImageNet) with a remaining-parameter comparison that depends on the chosen sparsity levels.
• The method still relies on a pruning-and-retraining pipeline, so the training cost is not reduced relative to standard LRR.

Experimental setup

Datasets: CIFAR-100 Tiny-ImageNet ImageNet

Benchmarks: CIFAR-100 Tiny-ImageNet ImageNet

Metrics: top-1 test accuracy SGD-noise stability (error barrier) linear mode connectivity error barrier

Baselines: LRR GM IMP Dense Network

Builds on: lottery ticket hypothesis iterative magnitude pruning learning rate rewinding linear mode connectivity GM (gradient-based signed mask search)

Closest papers by meaning

• The Elastic Lottery Ticket Hypothesis (NeurIPS 2021 · similarity 0.93)
• Dual Lottery Ticket Hypothesis (ICLR 2022 · similarity 0.93)
• Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets (ICML 2022 · similarity 0.92)
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (ICLR 2019 · similarity 0.92)
• Weisfeiler and Leman Go Gambling: Why Expressive Lottery Tickets Win (ICML 2025 · similarity 0.90)
• On the Sparsity of the Strong Lottery Ticket Hypothesis (NeurIPS 2024 · similarity 0.89)