Research map / Architectures and efficiency / Pruning and efficient tuning
Finding Lottery Tickets in Vision Models via Data-driven Spectral Foresight Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Path eXclusion (PX), a pruning-at-initialization method that scores weights by an analytical data-dependent upper bound on the NTK trace decomposed over network paths, and extends it to pruning pre-trained models for transfer.
Problem
Pruning at initialization (finding lottery-ticket-like sparse subnetworks before training) usually ignores the data-dependent component of the NTK spectrum, leading to poor sparse networks at high sparsity and layer collapse; it is also unclear whether pruning can be applied to pre-trained models be
Approach
Derives an analytical upper bound on the NTK trace that factorizes into a data-dependent path activation term and a path-kernel term, computable via two auxiliary networks and backpropagation; the resulting saliency is strictly positive, so with iterative global pruning it avoids layer collapse and preserves the data-aware NTK spectrum.
Key results
- On ImageNet ResNet-50 at 95.60% sparsity, PX reaches 60.28% accuracy versus 60.01% for GraSP and 59.84% for NTK-SAP.
- At 98.20% sparsity on VGG-16 (CIFAR-100), PX shows a considerable margin over all other methods including NTK-SAP, per the paper's figure-based comparison.
- On ResNet-20 CIFAR-10 at 98.20% sparsity with T=10 pruning rounds, PX achieves 77.04% accuracy, versus 74.55% for NTK-SAP with T=10 and 77.38% for IMP at 960 epochs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PX pruning | ResNet-50 · ImageNet at 89.26% sparsity | accuracy: 67.98 ± 0.31 -> 68.11 ± 0.29 | Helped |
| use PX pruning | ResNet-50 · ImageNet at 95.60% sparsity | accuracy: 59.84 ± 0.30 -> 60.28 ± 0.32 | Helped |
| use PX instead of GraSP | ResNet-20 · CIFAR-10 at 98.20% sparsity | accuracy: 76.30 -> 76.15 | Mixed |
| increase PX pruning rounds from T=1 to T=2 (ablation) | ResNet-20 · CIFAR-10 at 98.20% sparsity | accuracy: 76.15 -> 76.47 | Helped |
| increase PX pruning rounds from T=2 to T=10 (ablation) | ResNet-20 · CIFAR-10 at 98.20% sparsity | accuracy: 76.47 -> 77.04 | Helped |
| increase PX pruning rounds from T=10 to T=100 (ablation) | ResNet-20 · CIFAR-10 at 98.20% sparsity | accuracy: 77.04 -> 77.08 | Helped |
| increase PX pruning rounds from T=100 to T=200 (ablation) | ResNet-20 · CIFAR-10 at 98.20% sparsity | accuracy: 77.08 -> 77.39 | Helped |
| remove the data-dependent term from PX (ablation) | PX · CIFAR-10, ResNet-20 at 93.12% sparsity | Fixed-Weight-NTK eigenspectrum preservation | Hurt |
| use PX instead of NTK-SAP | ResNet-20 · CIFAR-10 at 98.20% sparsity | accuracy: 75.06 -> 77.39 | Helped |
Surprising findings
- NTK-SAP's performance drops drastically at extreme sparsity when initialized from pre-trained parameters, which the authors attribute to random mini-batches interfering with batch normalization statistics.
- Magnitude pruning outperforms SNIP on ImageNet ResNet-50 at 89.26% sparsity (66.56% vs 60.50%).
Limitations
- Segmentation results are reported only for PX, SNIP, Random and Magnitude because other methods failed (exploding gradients, layer collapse, or inapplicability).
- Current PaI methods only approximate dense network results at trivial sparsity levels on complex tasks such as segmentation, per the authors.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet Pascal VOC 2012
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet Pascal VOC 2012
Metrics: classification accuracy mean Intersection over Union execution time
Baselines: SynFlow SNIP GraSP NTK-SAP Random pruning Magnitude pruning IMP SynFlow-L2 LogSynFlow PHEW
Builds on: SynFlow NTK-SAP SNIP Lottery Ticket Hypothesis Neural Tangent Kernel
Closest papers by meaning
- Plant 'n' Seek: Can You Find the Winning Ticket? (ICLR 2022 · similarity 0.90)
- Winning the Lottery with Continuous Sparsification (NeurIPS 2020 · similarity 0.89)
- Validating the Lottery Ticket Hypothesis with Inertial Manifold Theory (NeurIPS 2021 · similarity 0.89)
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (ICLR 2019 · similarity 0.89)
- No Free Prune: Information-Theoretic Barriers to Pruning at Initialization (ICML 2024 · similarity 0.86)
- Fast Track to Winning Tickets: Repowering One-Shot Pruning for Graph Neural Networks (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.