Research map / Architectures and efficiency / Attention and transformers
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use deep self-attention distillation instead of the baseline distillation approaches | 6-layer, 768-hidden student distilled from BERT_BASE · SQuAD 2.0 dev | F1: 73.1 -> 76.4 | Helped |
| Add self-attention value-relation transfer (ablation) | MiniLM (6 layers, hidden size 384) · SQuAD 2.0 dev | F1: 71.0 -> 72.4 | Helped |
| Add self-attention value-relation transfer (ablation) | MiniLM (6 layers, hidden size 384) · MNLI-m dev | accuracy: 80.9 -> 82.2 | Helped |
| Add self-attention value-relation transfer (ablation) | MiniLM (6 layers, hidden size 384) · SST-2 dev | accuracy: 89.9 -> 91.0 | Helped |
| Replace value-relation KL loss with MSE over values (ablation) | MiniLM (6 layers, hidden size 384) · SQuAD 2.0 dev | F1: 71.4 -> 72.4 | Hurt |
| Replace value-relation KL loss with MSE over values (ablation) | MiniLM (6 layers, hidden size 384) · MNLI-m dev | accuracy: 82.0 -> 82.2 | Hurt |
| Replace value-relation KL loss with MSE over values (ablation) | MiniLM (6 layers, hidden size 384) · SST-2 dev | accuracy: 90.8 -> 91.0 | No clear effect |
| Add layer-to-layer distillation alongside last-layer distillation (ablation) | MiniLM (6 layers, hidden size 384) · SQuAD 2.0 dev | F1: 72.4 -> 71.6 | Hurt |
| Add layer-to-layer distillation alongside last-layer distillation (ablation) | MiniLM (6 layers, hidden size 384) · MNLI-m dev | accuracy: 82.2 -> 81.8 | Hurt |
| Add a teacher assistant | MiniLM (6 layers, hidden size 384) · SQuAD 2.0 dev | F1: 72.4 -> 72.7 | Helped |
| Add a teacher assistant | MiniLM (6 layers, hidden size 384) · MNLI-m dev | accuracy: 82.2 -> 82.4 | Helped |
| Add a teacher assistant | MiniLM (6 layers, hidden size 384) · SST-2 dev | accuracy: 91.0 -> 91.2 | Helped |
Closest papers by meaning
- How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.89)
- The Closeness of In-Context Learning and Weight Shifting for Softmax Regression (NeurIPS 2024 · similarity 0.89)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.89)
- Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers (NeurIPS 2024 · similarity 0.88)
- Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.88)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.