Research map / Large language models / Language, safety and interpretability
ExLM: Rethinking the Impact of $\texttt{[MASK]}$ Tokens in Masked Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes how [MASK] tokens corrupt context in MLMs and proposes ExLM, which expands each [MASK] into k states with a transition matrix and DAG-based state alignment, improving GLUE/SQuAD and MoleculeNet results.
Problem
MLM pre-training with [MASK] tokens introduces unreal tokens and corrupted semantics, with the latter (ambiguous, multimodal predictions) hurting downstream performance; existing work focused mainly on unreal tokens.
Approach
Duplicate each [MASK] embedding into k expanded states distinguished by 2D RoPE, model dependencies between states with an upper-triangular transition matrix (DAG), and train with a DP-based state alignment objective (DA-Transformer style) that marginalizes over alignments of states to target tokens.
Key results
- ExLM reaches 86.2 GLUE mean vs 85.2 for RoBERTa (same data/settings) and 82.0 EM / 84.6 F1 on SQuAD 2.0 vs RoBERTa's 78.3 / 81.5.
- On 7 MoleculeNet classification tasks, ExLM achieves the best mean ROC-AUC of 76.1 versus 74.2 for the next best (D-MPNN).
- Ablation on GLUE subsets: ExLM averages 81.4 vs 79.6 for Vanilla MLM; removing the transition matrix drops it to 80.4 and removing 2D RoPE to 80.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ExLM context expansion and dependency modeling instead of RoBERTa pre-training | BERT base-sized text MLM · GLUE dev | MEAN: 85.2 -> 86.2 | Helped |
| Add ExLM context expansion and dependency modeling instead of SMILES-BERT pre-training | SMILES-BERT · MoleculeNet scaffold split | Mean ROC-AUC | Helped |
| Remove 2D RoPE (ablation) | ExLM · GLUE dev ablation tasks | Avg: 81.4 -> 80.9 | Hurt |
| Remove the transition matrix (ablation) | ExLM · GLUE dev ablation tasks | Avg: 81.4 -> 80.4 | Hurt |
| Add expanded states and ExLM context modeling instead of Vanilla MLM | Vanilla MLM · GLUE dev ablation tasks | Avg: 79.6 -> 81.4 | Helped |
| Use ExLM rather than a same-training-cost Vanilla MLM++ | Vanilla MLM++ · GLUE dev ablation tasks | Avg: 80.7 -> 81.4 | Helped |
| Increase expanded states from k=2 to k=4 at p=15% (ablation) | ExLM · GLUE dev | Avg: 80.9 -> 81.4 | Helped |
| Increase expanded states from k=4 to k=8 at p=15% (ablation) | ExLM · GLUE dev | Avg: 81.4 -> 80.8 | Hurt |
| Increase expanded states from k=2 to k=4 at p=38.7% (ablation) | ExLM · GLUE dev | Avg: 80.5 -> 81.2 | Helped |
| Increase expanded states from k=4 to k=8 at p=38.7% (ablation) | ExLM · GLUE dev | Avg: 81.2 -> 80.7 | Hurt |
| Increase mask ratio from 15% to 38.7% (ablation) | Vanilla MLM · GLUE dev | Avg: 79.6 -> 79.2 | Hurt |
| Increase mask ratio from 15% to 38.7% (ablation) | ExLM-k=4 · GLUE dev | Avg: 81.4 -> 81.2 | Hurt |
Surprising findings
- Repeated-MLM experiments indicate corrupted semantics affects performance far more than the fraction of [MASK] tokens itself, with performance changing only slightly as mask ratio varies at fixed corruption.
- Too little corruption also hurts: the pre-training task becomes trivially easy and the model learns less.
Limitations
- Expansion increases sequence length and training cost; large k (e.g., 8) degrades performance.
- Gains on some MoleculeNet tasks are not uniform (e.g., BACE below GROVER).
Experimental setup
Datasets: English Wikipedia BookCorpus 19M-molecule SMILES dataset MNLI QQP QNLI SST-2 CoLA RTE MRPC STS-B SQuAD 2.0
Benchmarks: GLUE SQuAD 2.0 MoleculeNet MNLI QNLI QQP RTE
Metrics: accuracy F1 exact match ROC-AUC GLUE mean score
Baselines: BERT ALBERT XLNet UniLMv2 TUPE RoBERTa D-MPNN Attentive FP GROVER GraphMVP Mole-BERT 3D InfoMax
Builds on: BERT RoBERTa DA-Transformer ELECTRA connectionist temporal classification
Closest papers by meaning
- Representation Deficiency in Masked Language Modeling (ICLR 2024 · similarity 0.93)
- What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.87)
- SAMTok: Representing Any Mask with Two Words (CVPR 2026 · similarity 0.86)
- Layer by Layer: Uncovering Hidden Representations in Language Models (ICML 2025 · similarity 0.85)
- BERTs are Generative In-Context Learners (NeurIPS 2024 · similarity 0.85)
- Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration (AAAI 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.