Research map / Architectures and efficiency / Pruning and efficient tuning
DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MLM with RTD (DeBERTa+RTD, ES) in base analysis (ablation) | DeBERTa-base pre-training · MNLI-m / SQuAD v2.0 dev | MNLI-m accuracy and SQuAD F1: +2.5 MNLI-m, +3.8 SQuAD F1 (per text) | Helped |
| no embedding sharing (NES) between generator and discriminator (ablation) | DeBERTa+RTD base · MNLI-m / SQuAD v2.0 dev | MNLI-m accuracy and SQuAD F1: MNLI-m 88.3 vs 88.8 ES; SQuAD F1 85.3 vs 86.3 ES | Hurt |
| gradient-disentangled embedding sharing (GDES) instead of vanilla ES (ablation) | DeBERTa+RTD base · MNLI-m / SQuAD v2.0 dev | MNLI-m accuracy and SQuAD F1: MNLI-m 89.3 vs 88.8 ES; SQuAD F1 87.2 vs 86.3 ES | Helped |
| gradient-disentangled embedding sharing (GDES) instead of ES (ablation) | ELECTRA-base · MNLI-m / SQuAD v2.0 dev | MNLI-m accuracy and SQuAD F1: MNLI-m 88.3 vs 87.9 ES; SQuAD F1 85.9 vs 85.0 ES | Helped |
| no embedding sharing (NES) in ELECTRA-base (ablation) | ELECTRA-base · MNLI-m / SQuAD v2.0 dev | MNLI-m accuracy and SQuAD F1: MNLI-m 86.3 vs 87.9 ES; SQuAD F1 81.7 vs 85.0 ES | Hurt |
| DeBERTaV3 with RTD+GDES+DA vs DeBERTa-large (large scale) | DeBERTaV3-large · GLUE dev | average score: 91.37 vs 90.00 | Helped |
| multilingual RTD+GDES (mDeBERTaV3-base) vs XLM-R-base | mDeBERTaV3-base · XNLI cross-lingual transfer | average accuracy: +3.6 (79.8 vs 76.2) | Helped |
Closest papers by meaning
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.87)
- Leviathan: Decoupling Input and Output Representations in Language Models (NeurIPS 2026 · similarity 0.86)
- ShareBERT: Embeddings Are Capable of Learning Hidden Layers (AAAI 2024 · similarity 0.86)
- Fast-ELECTRA for Efficient Pre-training (ICLR 2024 · similarity 0.86)
- Large Language Model Unlearning via Embedding-Corrupted Prompts (NeurIPS 2024 · similarity 0.86)
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.