Research map / Large language models / Language, safety and interpretability
ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use continual multi-task learning instead of multi-task learning from scratch | ERNIE pre-training · MNLI, SST-2, MRPC | Fine-tuning result: 78.7 -> 79.0; 87.5 -> 87.8; 83.0 -> 84.0 | Helped |
| Use continual multi-task learning instead of sequential single-task continual learning | ERNIE pre-training · MNLI, SST-2, MRPC | Fine-tuning result: 77.3 -> 79.0; 86.4 -> 87.8; 82.5 -> 84.0 | Helped |
| Pre-train with ERNIE 2.0 rather than BERT LARGE | GLUE · GLUE test | Score: 80.5 -> 83.6 | Helped |
| Pre-train with ERNIE 2.0 LARGE rather than BERT BASE | Chinese NLP tasks · CMRC 2018 dev, DRCD test, DuReader dev, MSRA-NER test, XNLI test, ChnSentiCorp test, LCQMC test, BQ Corpus test, NLPCC-DBQA test | Reported task metrics | Helped |
Closest papers by meaning
- SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization (NeurIPS 2026 · similarity 0.86)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.86)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.86)
- SeqGPT: An Out-of-the-Box Large Language Model for Open Domain Sequence Understanding (AAAI 2024 · similarity 0.86)
- Analyzing Similarity Metrics for Data Selection for Language Model Pretraining (NeurIPS 2025 · similarity 0.86)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.