Research map / Architectures and efficiency / Attention and transformers
PMI-Masking: Principled masking of correlated spans
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Mask high-PMI correlated n-grams as joint units instead of selecting tokens randomly | BERT Base pretraining · SQuAD2.0, Wikipedia+BookCorpus | F1: 79.6 -> 81.4 | Helped |
| Mask high-PMI correlated n-grams as joint units instead of selecting tokens randomly | BERT Base pretraining · RACE, Wikipedia+BookCorpus | accuracy: 66.2 -> 68.4 | Helped |
| Use PMI-Masking rather than Random-Span Masking | BERT Base pretraining · RACE test, 2.4M steps, Wikipedia+BookCorpus | accuracy: 68.7 -> 70.9 | Helped |
| Use PMI-Masking rather than Random-Token Masking | BERT Base pretraining · RACE test, 2.4M steps, Wikipedia+BookCorpus+OpenWebText | accuracy: 73 -> 73.2 | Helped |
| Replace the proposed PMI_n ranking with Naive-PMI_n (ablation) | PMI-Masking · RACE test, 1M steps, Wikipedia+BookCorpus | accuracy: 68.4 -> 67.8 | Hurt |
| Replace the proposed PMI_n ranking with Naive-PMI_n (ablation) | PMI-Masking · SQuAD2.0 dev, 1M steps, Wikipedia+BookCorpus | F1: 81.4 -> 81.3 | No clear effect |
| Rank n-grams by frequency rather than PMI_n (ablation) | PMI-Masking · SQuAD2.0 dev, Wikipedia+BookCorpus checkpoints | F1: At 1M steps: 79.7 -> 81.3 | Hurt |
| Use Whole-Word Masking instead of Random-Token Masking (ablation) | BERT Base pretraining · SQuAD2.0 dev, 2K vocabulary | F1: 72.8 -> 77.6 | Helped |
| Use Whole-Word Masking instead of Random-Token Masking (ablation) | BERT Base pretraining · SQuAD2.0 dev, 30K vocabulary | F1: 79.3 -> 79.7 | No clear effect |
| Increase pretraining vocabulary size from 30K to 2K tokens (ablation) | Random-Token Masking BERT Base · SQuAD2.0 dev | F1: 79.3 -> 72.8 | Hurt |
| Increase pretraining data by adding OpenWebText | PMI-Masking BERT Base · SQuAD2.0 test, 2.4M steps | F1: 83.6 -> 84.9 | Helped |
| Train with PMI-Masking rather than Random-Span Masking | BERT Base pretraining · SQuAD2.0 dev, 2.4M steps, Wikipedia+BookCorpus+OpenWebText | F1: 83.3 -> 84.9 | Helped |
Closest papers by meaning
- Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning (ICML 2024 · similarity 0.89)
- Representation Deficiency in Masked Language Modeling (ICLR 2024 · similarity 0.88)
- Masking meets Supervision: A Strong Learning Alliance (CVPR 2025 · similarity 0.87)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.86)
- Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More (ICML 2025 · similarity 0.85)
- ExLM: Rethinking the Impact of $\texttt{[MASK]}$ Tokens in Masked Language Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.