Research map / Architectures and efficiency / Attention and transformers
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ALBERT-xxlarge instead of BERT-large | Pretrained language model · RACE dev | accuracy: 73.9 -> 82.3 | Helped |
| Use ALBERT-xxlarge instead of BERT-large | Pretrained language model · Downstream task average dev | Avg: 85.2 -> 88.7 | Helped |
| Share all parameters across layers rather than sharing none (ablation) | ALBERT-base, E=128 · Downstream task average | Avg: 81.6 -> 80.1 | Hurt |
| Share attention parameters only rather than sharing none (ablation) | ALBERT-base, E=128 · Downstream task average | Avg: 81.6 -> 81.7 | Helped |
| Share FFN parameters only rather than sharing none (ablation) | ALBERT-base, E=128 · Downstream task average | Avg: 81.6 -> 80.2 | Hurt |
| Use embedding size 128 instead of 64, 256, or 768 (ablation) | ALBERT-base with all parameters shared · Downstream task average | Avg: 79.0; 80.1; 79.6; 79.8 | Helped |
| Use SOP rather than no inter-sentence loss (ablation) | ALBERT-base · Downstream task average | Avg: 79.0 -> 80.1 | Helped |
| Use SOP rather than NSP (ablation) | ALBERT-base · Downstream task average | Avg: 79.2 -> 80.1 | Helped |
| Add additional pretraining data (ablation) | ALBERT-base · SQuAD 1.1 dev | F1/EM: 89.3/82.3 -> 88.8/81.7 | Hurt |
| Add additional pretraining data (ablation) | ALBERT-base · Downstream task average | Avg: 80.1 -> 80.8 | Helped |
| Remove dropout during training (ablation) | ALBERT-xxlarge · Downstream task average | Avg: 90.4 -> 90.7 | Helped |
| Increase depth from 12 to 24 layers (ablation) | ALBERT-xxlarge · Downstream task average | Avg: 88.7 -> 88.7 | No clear effect |
Closest papers by meaning
- PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination (ICML 2020 · similarity 0.88)
- Incorporating BERT into Parallel Sequence Decoding with Adapters (NeurIPS 2020 · similarity 0.87)
- MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining (NeurIPS 2023 · similarity 0.87)
- Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study (ICLR 2023 · similarity 0.87)
- I-BERT: Integer-only BERT Quantization (ICML 2021 · similarity 0.87)
- ShareBERT: Embeddings Are Capable of Learning Hidden Layers (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.