Research map / Large language models / Language, safety and interpretability
Representation Deficiency in Masked Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that MLM pretraining reserves model dimensions for [MASK] tokens, causing rank deficiency for real-token representations, and proposes MAE-LM, which excludes [MASK] from the encoder and uses a shallow decoder, improving GLUE and SQuAD 2.0 results over MLM baselines.
Problem
MLM pretraining introduces a [MASK] token absent from downstream data, which may cause the learned representation space for real tokens to be rank-deficient, limiting the pretrained encoder's expressiveness when fine-tuned on real text.
Approach
Identify representation deficiency theoretically and empirically (effective rank of real-token representations is lower than with [MASK] inputs), then pretrain a Masked Autoencoder-style encoder that omits [MASK] tokens entirely, with an auxiliary bidirectional decoder that receives [MASK] positions and is discarded after pretraining.
Key results
- On the base setting, MAE-LM achieves a GLUE average of 86.1 vs 85.2 for RoBERTa (Ours), and SQuAD 2.0 EM/F1 of 81.1/84.1 vs 78.3/81.5 for RoBERTa (Ours).
- In the base++ setting, MAE-LM reaches a GLUE average of 87.8 and SQuAD 2.0 EM/F1 of 83.5/86.5, outperforming RoBERTa (86.4 GLUE; 80.5/83.7 SQuAD).
- MAE-LM reaches RoBERTa's MNLI accuracy with about 40% of its pretraining time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| exclude [MASK] tokens from the encoder during pretraining | MAE-LM vs. RoBERTa (Ours) · GLUE, base | AVG: 85.2 -> 86.1 | Helped |
| use MAE-LM pretraining instead of UniLMv2 | 12-layer base++ models · GLUE, base++ | AVG: 87.1 -> 87.8 | Helped |
| add the MAE-LM decoder while retaining [MASK] tokens in the encoder (ablation) | standard MLM · GLUE, base | AVG: 85.2 -> 85.1 | Hurt |
| include [MASK] tokens in the encoder and reset their decoder embeddings (ablation) | MAE-LM · GLUE, base | AVG: 86.1 -> 85.9 | Hurt |
| replace encoder [MASK] tokens with random real tokens (ablation) | MAE-LM · GLUE, base | AVG: 86.1 -> 85.1 | Hurt |
| use misaligned consecutive position encoding (ablation) | MAE-LM · GLUE, base | AVG: 86.1 -> 86.0 | No clear effect |
| remove relative position encoding (ablation) | MAE-LM · GLUE, base | AVG: 86.1 -> 86.1 | No clear effect |
| add cross-attention to bidirectional decoder self-attention (ablation) | MAE-LM · GLUE, base | AVG: 86.1 -> 85.4 | Hurt |
| use unidirectional decoder self-attention and cross-attention (ablation) | MAE-LM · GLUE, base | AVG: 86.1 -> 85.5 | Hurt |
| use cross-attention only in the decoder (ablation) | MAE-LM · GLUE, base | AVG: 86.1 -> 86.0 | No clear effect |
| reduce decoder depth from 4 to 2 layers (ablation) | MAE-LM · GLUE, base | AVG: 86.1 -> 85.8 | Hurt |
| increase decoder depth from 4 to 6 layers (ablation) | MAE-LM · GLUE, base | AVG: 86.1 -> 84.8 | Hurt |
Surprising findings
- Simply adding the same decoder to standard MLM does not improve downstream performance (GLUE 85.1 vs 85.2).
- Replacing [MASK] tokens with random real tokens does not beat standard MLM despite avoiding representation deficiency.
Limitations
- Analysis and method apply only to MLM-style objectives using [MASK] tokens; not to autoregressive or other pretraining settings.
- Empirical comparisons mainly against MLM and simple variants rather than state-of-the-art models that combine multiple strategies or architectures.
- Theoretical analysis simplifies by not accounting for MLPs and residual connections (single-head attention in proofs).
Experimental setup
Datasets: Wikipedia BookCorpus OpenWebText CC-News STORIES GLUE SQuAD 2.0
Benchmarks: GLUE SQuAD 2.0
Metrics: accuracy Matthews correlation Spearman correlation exact match F1 effective rank
Baselines: BERT ALBERT UniLMv2 TUPE RoBERTa
Builds on: masked language modeling RoBERTa Masked Autoencoder BERT
Closest papers by meaning
- ExLM: Rethinking the Impact of $\texttt{[MASK]}$ Tokens in Masked Language Models (ICML 2025 · similarity 0.93)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.86)
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.86)
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.86)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.86)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.