Research map / Architectures and efficiency / Attention and transformers
MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MosaicBERT architecture and recipe instead of standard BERT-Base | BERT-Base · GLUE dev avg, C4 pretraining | time to reach avg GLUE 83.2 (hours): 11.5 -> 4.6 | Helped |
| Use MosaicBERT-Large instead of BERT-Large | BERT-Large · GLUE dev avg, 2 epochs C4 | time to reach avg GLUE 83.2 (hours): 23.35 -> 15.85 | Helped |
| Raise MLM masking ratio from 15% to 30% (ablation) | BERT-Base · GLUE dev avg | accuracy and wall clock time | Helped |
| Set attention dropout to 0 (BERT+drpt=0) (ablation) | BERT-Base · GLUE dev avg | accuracy and wall clock time | Helped |
| Low precision (bfloat16) LayerNorm (ablation) | BERT-Base · GLUE dev avg | pretraining speed | Helped |
| Add Gated Linear Units to feedforward layers (ablation) | BERT-Base · GLUE dev avg | accuracy vs wall clock time | Mixed |
| Remove FlashAttention and set attention dropout 0.1 from MosaicBERT (ablation) | MosaicBERT-Base · GLUE dev avg | wall clock time and accuracy | No clear effect |
| Use MosaicBERT-Base instead of BERT-Base (178k steps) | BERT-Base · GLUE dev avg | average GLUE: 84.7 -> 85.4 | Helped |
| Use MosaicBERT-Large instead of BERT-Large (178k steps) | BERT-Large · GLUE dev avg | average GLUE: 85.2 -> 86.1 | Helped |
| Use MosaicBERT-Base instead of BERT-Base | BERT-Base · GLUE dev RTE | accuracy: 78.2 -> 83 | Helped |
| Use MosaicBERT-Base instead of BERT-Base | BERT-Base · GLUE dev CoLA | score: 59 -> 58.2 | No clear effect |
| Scale from MosaicBERT-Base to MosaicBERT-Large | MosaicBERT-Base · GLUE dev avg | average GLUE after long training: 85.5 | Helped |
Closest papers by meaning
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (ICLR 2020 · similarity 0.88)
- AutoBERT-Zero: Evolving BERT Backbone from Scratch (AAAI 2022 · similarity 0.88)
- BiBERT: Accurate Fully Binarized BERT (ICLR 2022 · similarity 0.88)
- DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION (ICLR 2021 · similarity 0.88)
- MPNet: Masked and Permuted Pre-training for Language Understanding (NeurIPS 2020 · similarity 0.88)
- No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.