Research map / Architectures and efficiency / Attention and transformers
DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use disentangled attention with content-to-position and position-to-content terms | DeBERTa-base · RACE development set | accuracy: 66.7 -> 71.7 | Helped |
| Add an enhanced mask decoder with absolute positions (ablation) | DeBERTa-base · RACE development set | accuracy: 70.3 -> 71.7 | Helped |
| Add an enhanced mask decoder with absolute positions (ablation) | DeBERTa-base · SQuAD v2.0 development set | F1: 81.3 -> 82.5 | Helped |
| Add the content-to-position attention term (ablation) | DeBERTa-base · RACE development set | accuracy: 69.3 -> 71.7 | Helped |
| Add the position-to-content attention term (ablation) | DeBERTa-base · RACE development set | accuracy: 69.6 -> 71.7 | Helped |
| Add both EMD and content-to-position (ablation) | DeBERTa-base · RACE development set | accuracy: 68.1 -> 71.7 | Helped |
| Add both EMD and position-to-content (ablation) | DeBERTa-base · RACE development set | accuracy: 68.5 -> 71.7 | Helped |
| Place absolute position embeddings in the decoder rather than the input layer (ablation) | DeBERTa · Wikitext-103 test | perplexity: 20.0 -> 19.9 | Helped |
| Jointly train on MLM and ARLM (ablation) | DeBERTa · Wikitext-103 test | perplexity: 19.9 -> 19.5 | Helped |
| Share relative-position and content projection matrices (ablation) | DeBERTa-base · MNLI-m/mm development set | accuracy: 86.3/86.2 -> 86.3/86.3 | No clear effect |
| Add a convolution layer beside the first Transformer layer (ablation) | DeBERTa-base · SQuAD v1.1 development set | F1: 92.1 -> 92.5 | Helped |
| Use a 128k vocabulary (ablation) | DeBERTa-base+Conv · MNLI-m/mm development set | accuracy: 86.3/86.5 -> 86.7/86.9 | Helped |
Closest papers by meaning
- DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing (ICLR 2023 · similarity 0.89)
- MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining (NeurIPS 2023 · similarity 0.88)
- DynaBERT: Dynamic BERT with Adaptive Width and Depth (NeurIPS 2020 · similarity 0.88)
- BiBERT: Accurate Fully Binarized BERT (ICLR 2022 · similarity 0.87)
- InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective (ICLR 2021 · similarity 0.87)
- AutoBERT-Zero: Evolving BERT Backbone from Scratch (AAAI 2022 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.