Research map / Large language models / Language, safety and interpretability
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, Hsiao-Wuen Hon
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly train AE and PAR rather than using either objective alone (ablation) | UniLMv2 · SQuAD v1.1 | F1: 91.0 -> 92.0; 90.7 -> 92.0 | Improved |
| Jointly train AE and PAR rather than using AR alone (ablation) | UniLMv2 · SQuAD v1.1 | F1: 89.9 -> 92.0 | Improved |
| Use PAR rather than AR for pre-training (ablation) | UniLMv2 · SST-2 | accuracy: 92.3 -> 92.5 | Improved |
| Add blockwise factorization to AR-only pre-training (ablation) | UniLMv2 · SQuAD v1.1 | F1: 89.9 -> 90.7 | Improved |
| Add relative position bias (ablation) | UniLMv2 · SQuAD v1.1 | F1: 91.5 -> 92.0 | Improved |
| Add relative position bias (ablation) | UniLMv2 · XSum | ROUGE-1: 43.69 -> 44.00 | Mixed |
| Add relative position bias (ablation) | UniLMv2 · Question generation, first split | BLEU-4: 24.70 -> 24.43 | No clear effect |
| Use UniLMv2 instead of RoBERTa base | SQuAD v1.1 · SQuAD v1.1 | F1: 90.6 -> 92.0 | Improved |
| Use UniLMv2 instead of BART large | XSum · XSum | ROUGE-1: 45.14 -> 44.00 | Hurt |
Later papers that compared with this method
1 later accepted paper reports UniLM V2 Base++ as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (NeurIPS 2021 · better on 1 of 1 shared result)
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.
- XLNet base — XLNet: Generalized Autoregressive Pretraining for Language Understanding (NeurIPS 2019) · better on 2 of 3 shared results
- MASS base — MASS: Masked Sequence to Sequence Pre-training for Language Generation (ICML 2019) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More (ICML 2025 · similarity 0.88)
- Representation Deficiency in Masked Language Modeling (ICLR 2024 · similarity 0.88)
- Reconsidering Positional Supervision in Masked Diffusion Language Model Training (NeurIPS 2026 · similarity 0.86)
- MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models (NeurIPS 2024 · similarity 0.86)
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.86)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
