Research map / Architectures and efficiency / Attention and transformers
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Progressively eliminate word-vectors using attention-based significance scores | BERT_BASE · CoLA, RTE, QQP, MRPC, SST-2, MNLI-m, QNLI and STS-B | inference time: (4.5x), (3.4x), (4.5x), (2.7x), (2.4x), (2.6x), (2.0x), (2.0x) speedup | Helped |
| Progressively eliminate word-vectors using attention-based significance scores | BERT_BASE · CoLA, RTE, QQP, MRPC, SST-2, MNLI-m, QNLI and STS-B | task score: 52.5 -> 52.3; 68.1 -> 67.4; 71.2 -> 70.2; 88.7 -> 88.1; 93.0 -> 92.1; 84.6 -> 83.8; 91.0 -> 90.1; 85.8 -> 85.1 | Hurt |
| Select vectors dynamically with attention significance rather than static head positions (ablation) | PoWER-BERT word-vector selection · SST-2 Dev, entire dataset | accuracy: 85.4% -> 88.3% | Helped |
| Select vectors dynamically with attention significance rather than random static positions (ablation) | PoWER-BERT word-vector selection · SST-2 Dev, entire dataset | accuracy: 85.7% -> 88.3% | Helped |
| Select vectors dynamically with attention significance rather than static head positions (ablation) | PoWER-BERT word-vector selection · SST-2 Dev, input sequence length > 16 | accuracy: 83.7% -> 87.4% | Helped |
| Select vectors dynamically with attention significance rather than random static positions (ablation) | PoWER-BERT word-vector selection · SST-2 Dev, input sequence length > 16 | accuracy: 83.4% -> 87.4% | Helped |
Closest papers by meaning
- MA-BERT: Towards Matrix Arithmetic-only BERT Inference by Eliminating Complex Non-Linear Functions (ICLR 2023 · similarity 0.89)
- I-BERT: Integer-only BERT Quantization (ICML 2021 · similarity 0.89)
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (ICLR 2020 · similarity 0.88)
- BERT Loses Patience: Fast and Robust Inference with Early Exit (NeurIPS 2020 · similarity 0.88)
- Incorporating BERT into Parallel Sequence Decoding with Adapters (NeurIPS 2020 · similarity 0.87)
- Incorporating BERT into Neural Machine Translation (ICLR 2020 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.