Research map / Architectures and efficiency / Attention and transformers
Efficient Training of BERT by Progressively Stacking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train BERT with progressive stacking (shallow to deep) instead of from scratch | BERT-Base · Pre-training wall time | hours: 441 -> 338 | Helped |
| Progressive stacking reaches baseline validation loss | BERT-Base · Pre-training validation loss | training time fraction: 76.77% | Helped |
| Stacking vs baseline on CoLA | BERT-Base · GLUE test CoLA | Matthews corr: 52.8 -> 56.2 | Helped |
| Stacking vs baseline on overall GLUE | BERT-Base · GLUE test | GLUE score: 77.4 -> 78.4 | Helped |
| Stacking vs baseline on STS-B | BERT-Base · GLUE test STS-B | Pearson/Spearman: 81.2/80.0 -> 84.2/82.5 | Helped |
| Stacking vs baseline on RTE | BERT-Base · GLUE test RTE | accuracy: 64.9 -> 67.0 | Helped |
| Stacking vs baseline on QNLI | BERT-Base · GLUE test QNLI | accuracy: 90.4 -> 90.1 | No clear effect |
| Stacking vs baseline on MNLI-m | BERT-Base · GLUE test MNLI-m | accuracy: 84.4 -> 84.4 | No clear effect |
| Stacking vs baseline on MRPC | BERT-Base · GLUE test MRPC | F1/accuracy: 87.3/83.0 -> 88.2/83.9 | Helped |
| Replace stacking initialization with identity or identity+noise function-preserving initialization | progressive training · Pre-training validation loss | validation loss | Hurt |
| Switch time t before threshold theta in 3-to-6 and 6-to-12 stacking (ablation) | progressive stacking · Pre-training validation loss | speedup | Helped |
| Switch time t after threshold theta (ablation) | progressive stacking · Pre-training validation loss | speedup | No clear effect |
Closest papers by meaning
- AutoBERT-Zero: Evolving BERT Backbone from Scratch (AAAI 2022 · similarity 0.88)
- MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining (NeurIPS 2023 · similarity 0.88)
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (ICLR 2020 · similarity 0.86)
- I-BERT: Integer-only BERT Quantization (ICML 2021 · similarity 0.86)
- MA-BERT: Towards Matrix Arithmetic-only BERT Inference by Eliminating Complex Non-Linear Functions (ICLR 2023 · similarity 0.86)
- No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.