Research map / Architectures and efficiency / Attention and transformers
The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms
Hikari Otsuka, Daiki Chijiwa, Yasuyuki Okoshi, Daichi Fujiki, Susumu Takeuchi, Masato Motomura
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add query/key weight scaling by n_K^(1/4) at initialization (ablation) | SLT search with edge-popup in GPT-2 · WikiText-103 | validation loss | Improved |
| increase query/key weight scale beyond n_K^(1/4) (ablation) | SLT search with edge-popup in GPT-2 · WikiText-103 | validation loss | Hurt |
| increase source hidden dimension n_K | SLT in random MHA · Synthetic angular velocity | approximation error: epsilon = 0.8 exp(-0.06 n_K) | Improved |
| increase input sequence length T | SLT in random MHA · Synthetic angular velocity | approximation error | No clear effect |
| increase number of transformer blocks with shared decay rate | SLT in random transformer · Synthetic angular velocity | approximation error | Improved |
Closest papers by meaning
- Polynomially Over-Parameterized Convolutional Neural Networks Contain Structured Strong Winning Lottery Tickets (NeurIPS 2023 · similarity 0.91)
- A General Framework For Proving The Equivariant Strong Lottery Ticket Hypothesis (ICLR 2023 · similarity 0.90)
- On the Sparsity of the Strong Lottery Ticket Hypothesis (NeurIPS 2024 · similarity 0.90)
- Convolutional and Residual Networks Provably Contain Lottery Tickets (ICML 2022 · similarity 0.90)
- Proving the Lottery Ticket Hypothesis for Convolutional Neural Networks (ICLR 2022 · similarity 0.90)
- Weisfeiler and Leman Go Gambling: Why Expressive Lottery Tickets Win (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
