Research map / Architectures and efficiency / Neural architecture search
AutoBERT-Zero: Evolving BERT Backbone from Scratch
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the BERT-base backbone with the searched hybrid AutoBERT-Zero backbone | BERT-base · GLUE dev | AVG: 83.1 -> 86.3 | Helped |
| replace the BERT-base backbone with the searched hybrid AutoBERT-Zero backbone | BERT-base · SQuAD v1.1 | F1: 86.7 -> 87.6 | Helped |
| replace the BERT-base backbone with the searched hybrid AutoBERT-Zero backbone | BERT-base · SQuAD v2.0 | F1: 72.8 -> 75.0 | Helped |
| use a hybrid of searched convolution and attention blocks instead of attention-only layers (ablation) | AutoBERT-Zero · GLUE dev | AVG: 84.8 -> 86.3 | Helped |
| use a hybrid of searched convolution and attention blocks instead of convolution-only layers (ablation) | AutoBERT-Zero · GLUE dev | AVG: 73.3 -> 86.3 | Helped |
| use the descending convolution-kernel-size pattern (ablation) | AutoBERT-Zero · GLUE dev | AVG: 85.6 -> 86.3 | Helped |
| use the hybrid searched backbone rather than three continuous attention layers (ablation) | AutoBERT-Zero · GLUE dev | AVG: 85.9 -> 86.3 | Helped |
| use the hybrid searched backbone rather than three continuous attention layers (ablation) | AutoBERT-Zero · SQuAD v2.0 | F1: 75.7 -> 75.0 | Hurt |
| scale AutoBERT-Zero from base to large configuration | AutoBERT-Zero · GLUE | AVG: 84.5 vs 82.1 | Helped |
| use the AutoBERT-Zero-small backbone instead of BERT-small | BERT-small · GLUE | AVG: 75.1 -> 80.5 | Helped |
| initialize AutoBERT-Zero from the supernet (ablation) | AutoBERT-Zero · GLUE test | AVG: 82.2 -> 83.5 | Helped |
Closest papers by meaning
- MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining (NeurIPS 2023 · similarity 0.88)
- Efficient Training of BERT by Progressively Stacking (ICML 2019 · similarity 0.88)
- MA-BERT: Towards Matrix Arithmetic-only BERT Inference by Eliminating Complex Non-Linear Functions (ICLR 2023 · similarity 0.88)
- Incorporating BERT into Parallel Sequence Decoding with Adapters (NeurIPS 2020 · similarity 0.87)
- BERT Loses Patience: Fast and Robust Inference with Early Exit (NeurIPS 2020 · similarity 0.87)
- ShareBERT: Embeddings Are Capable of Learning Hidden Layers (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.