Research map / Architectures and efficiency / Pruning and efficient tuning
The Lottery Ticket Hypothesis for Pre-trained BERT Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use IMP mask at pre-trained init instead of random pruning mask (ablation) | BERT-base subnetwork · MNLI 70% sparsity | matched accuracy: 67.5 -> 82.6 | Helped |
| use pre-trained init instead of random init for IMP subnetwork (ablation) | BERT-base subnetwork · MNLI 70% sparsity | matched accuracy: 61.0 -> 82.6 | Helped |
| use pre-trained init instead of layer-shuffled pre-trained weights (ablation) | BERT-base subnetwork · MNLI 70% sparsity | matched accuracy: 70.1 -> 82.6 | Helped |
| use IMP mask instead of random pruning mask (ablation) | BERT-base subnetwork · CoLA 50% sparsity | Matthew's correlation: 9.6 -> 53.8 | Helped |
| use IMP mask instead of random pruning mask (ablation) | BERT-base subnetwork · SQuAD 40% sparsity | F1: 31.8 -> 87.7 | Helped |
| rewind IMP subnetwork to later training step instead of pre-trained init (ablation) | BERT-base subnetwork · GLUE/SQuAD tasks | accuracy/F1 | No clear effect |
| rewind IMP subnetwork to 5% of training (ablation) | BERT-base subnetwork · RTE 60% sparsity | accuracy: 66.06 -> 62.09 | Hurt |
| use standard pruning instead of IMP at pre-trained init (ablation) | BERT-base subnetwork · SST-2 60% sparsity | accuracy: 91.74 -> 90.02 | Hurt |
| use standard pruning instead of IMP at pre-trained init (ablation) | BERT-base subnetwork · QNLI 70% sparsity | accuracy: 88.05 -> 89.88 | Helped |
| find the subnetwork on MLM instead of a downstream task, then transfer | 70% sparse BERT subnetwork · transfer across 11 tasks | transfer performance vs same-task | Helped |
| prune pre-trained weights directly (no iterative MLM IMP) (ablation) | MLM subnetwork · transfer across tasks | transfer performance | Hurt |
| rewind SQuAD subnetwork to later step before transfer instead of theta_0 (ablation) | SQuAD subnetwork · transfer to target tasks | target task performance | No clear effect |
Closest papers by meaning
- Playing Lottery Tickets with Vision and Language (AAAI 2022 · similarity 0.91)
- Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP (ICLR 2020 · similarity 0.90)
- The Lottery Tickets Hypothesis for Supervised and Self-Supervised Pre-Training in Computer Vision Models (CVPR 2021 · similarity 0.90)
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (ICLR 2019 · similarity 0.89)
- Efficient stagewise pretraining via progressive subnetworks (ICLR 2025 · similarity 0.87)
- Weisfeiler and Leman Go Gambling: Why Expressive Lottery Tickets Win (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.