Research map / Large language models / Language, safety and interpretability
MASS: Masked Sequence to Sequence Pre-training for Language Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MASS pre-training instead of XLM pre-training | 6-layer Transformer · WMT newstest2014 en-fr, unsupervised NMT | BLEU: 33.40 -> 37.50 | Helped |
| use MASS pre-training instead of XLM pre-training | 6-layer Transformer · unsupervised NMT, six WMT translation directions | BLEU | Helped |
| use MASS pre-training instead of BERT+LM or DAE pre-training | unsupervised NMT · six unsupervised translation pairs | BLEU | Helped |
| use MASS pre-training instead of training only on paired data | low-resource NMT · WMT English-French, English-German, and English-Romanian; 10K, 100K, and 1M pairs | BLEU | Helped |
| use MASS pre-training instead of training only on paired data | text summarization · Gigaword, 10K and 100K training pairs | ROUGE: more than 10 ROUGE points gain on 10K data; 5 ROUGE points gain on 100K data | Helped |
| use MASS pre-training instead of BERT+LM or DAE pre-training | text summarization · Gigaword, 3.8M training pairs | ROUGE-1 F1: 37.75 -> 38.73; 35.97 -> 38.73 | Helped |
| use MASS pre-training instead of training on available dialog pairs only | conversational response generation · Cornell Movie Dialog, 10K training pairs | perplexity: 82.39 -> 74.32 | Helped |
| use MASS pre-training instead of training on available dialog pairs only | conversational response generation · Cornell Movie Dialog, 110K training pairs | perplexity: 26.38 -> 23.52 | Helped |
| use a masked fragment near 50% of sentence length rather than the extreme settings k=1 or k=m (ablation) | MASS · unsupervised NMT, summarization, and conversational response generation | task performance | Helped |
| mask consecutive tokens rather than randomly selected discrete tokens (ablation) | MASS · WMT newstest2014 en-fr, unsupervised NMT | BLEU: 36.9 -> 37.5 | Helped |
| mask decoder inputs that are unmasked on the encoder side rather than feeding all tokens to the decoder (ablation) | MASS · WMT newstest2014 en-fr, unsupervised NMT | BLEU: 35.3 -> 37.5 | Helped |
Closest papers by meaning
- Incorporating BERT into Parallel Sequence Decoding with Adapters (NeurIPS 2020 · similarity 0.89)
- Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings (ICLR 2022 · similarity 0.88)
- Masked Structural Growth for 2x Faster Language Model Pre-training (ICLR 2024 · similarity 0.85)
- Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More (ICML 2025 · similarity 0.85)
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.85)
- Representation Deficiency in Masked Language Modeling (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.