Research map / Architectures and efficiency / RNNs and normalization
DynaBERT: Dynamic BERT with Adaptive Width and Depth
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train sub-networks jointly in one width-adaptive model instead of separate networks (ablation) | BERT-base · GLUE avg | accuracy: 77.6 -> 79.0 | Helped |
| add network rewiring by head/neuron importance (ablation) | DynaBERT_W · GLUE avg | accuracy: 79.0 -> 81.4 | Helped |
| add knowledge distillation and data augmentation (ablation) | DynaBERT_W · GLUE avg | accuracy: 81.4 -> 82.9 | Helped |
| use DynaBERT_W as teacher assistant vs direct distillation (ablation) | DynaBERT · SST-2, CoLA, MRPC | accuracy | Helped |
| width-adaptive first instead of depth-adaptive first (ablation) | DynaBERT · GLUE CoLA dev 0.5x | Matthews corr: 52.2 -> 56.7 | Helped |
| final fine-tuning on original data (ablation) | DynaBERT · SST-2, CoLA, MRPC | accuracy | Mixed |
| inplace distillation instead of conventional (ablation) | DynaBERT · SST-2, CoLA, MRPC, RTE | accuracy | Mixed |
| progressive rewiring or universally slimmable training (ablation) | DynaBERT_W · GLUE avg | accuracy: 82.9 -> 81.6 | No clear effect |
| compress width vs depth by same multiplier | DynaBERT · GLUE dev | accuracy | Helped |
Closest papers by meaning
- DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION (ICLR 2021 · similarity 0.88)
- BiBERT: Accurate Fully Binarized BERT (ICLR 2022 · similarity 0.87)
- MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining (NeurIPS 2023 · similarity 0.87)
- AutoBERT-Zero: Evolving BERT Backbone from Scratch (AAAI 2022 · similarity 0.87)
- ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques (AAAI 2021 · similarity 0.87)
- ShareBERT: Embeddings Are Capable of Learning Hidden Layers (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.