Research map / Label-efficient and robust learning / Recognition and long-tail
Cross-lingual Language Model Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TLM to MLM pretraining (ablation) | XLM · XNLI | average accuracy: 71.5\% to 75.1\%; 3.6\% accuracy | Helped |
| Fine-tune on translated training data in each XNLI language | XLM MLM+TLM · XNLI | average accuracy: 1.6\% | Helped |
| Use MLM rather than CLM pretraining (ablation) | Unsupervised MT · WMT'14 English-French | BLEU: 30.4 to 33.4 BLEU | Helped |
| Use MLM rather than CLM pretraining (ablation) | Unsupervised MT · WMT'16 Romanian-English | BLEU: 28.0 to 31.8 | Helped |
| Pretrain both encoder and decoder with MLM instead of only the lookup table (ablation) | Unsupervised MT · WMT'16 German-English | BLEU: up to 7 BLEU | Helped |
| Pretrain only the decoder instead of both encoder and decoder (ablation) | Unsupervised MT · Machine translation benchmarks | BLEU | Hurt |
| Pretrain only the encoder instead of both encoder and decoder (ablation) | Unsupervised MT · Machine translation benchmarks | BLEU | Hurt |
| Use MLM rather than CLM pretraining (ablation) | Supervised MT · WMT'16 Romanian-English | BLEU | Helped |
| Add cross-lingual language model pretraining (ablation) | Supervised MT · WMT'16 Romanian-English | BLEU | Helped |
| Add English data to Nepali language modeling (ablation) | Nepali language model · Nepali language modeling | perplexity: 157.2 to 140.1; 17.1 points | Helped |
| Add Hindi data to Nepali language modeling (ablation) | Nepali language model · Nepali language modeling | perplexity: reduction of 41.6 | Helped |
| Add both English and Hindi data (ablation) | Nepali language model · Nepali language modeling | perplexity: 109.3 | Helped |
Closest papers by meaning
- XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge (AAAI 2022 · similarity 0.92)
- Alternating Language Modeling for Cross-Lingual Pre-Training (AAAI 2020 · similarity 0.91)
- Cross-Lingual Natural Language Generation via Pre-Training (AAAI 2020 · similarity 0.90)
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation (ICML 2020 · similarity 0.89)
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.87)
- Constrained Decoding for Cross-lingual Label Projection (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.