Research map / Large language models / Language, safety and interpretability
XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Masked Entity Prediction and Object Entailment to multilingual pre-training | XLM-R_base · MLQA | F1: outperforming the baseline model XLM-R_base by 2.0 | Helped |
| Add Masked Entity Prediction and Object Entailment to multilingual pre-training | XLM-R_base · NER | F1: improves the Avg score to 79.8 | Helped |
| Add Masked Entity Prediction and Object Entailment to multilingual pre-training | XLM-R_base · XNLI | test accuracy: 0.6 gain | Helped |
| Add Object Entailment to the MEP-only model (ablation) | XLM-K pre-training · MLQA | F1: 0.7 | Helped |
| Add Masked Entity Prediction to the OE-only model (ablation) | XLM-K pre-training · MLQA | F1: 0.4 | Helped |
| Add both knowledge-task losses instead of MLM on multilingual Wikipedia entity descriptions (ablation) | XLM-K pre-training · MLQA | F1: drops by 1.5 | Helped |
| Add both knowledge-task losses instead of MLM on multilingual Wikipedia entity descriptions (ablation) | XLM-K pre-training · NER | F1: drops by 0.8 | Helped |
| Add both knowledge-task losses instead of MLM on multilingual Wikipedia entity descriptions (ablation) | XLM-K pre-training · XNLI | test accuracy: drops by 0.3 | Helped |
| Use multilingual entity descriptions in MEP (ablation) | MEP · MLQA | F1: drops by 0.5 | Helped |
| Use multilingual entity descriptions in MEP (ablation) | MEP · NER | F1: drops by 0.3 | Helped |
| Use multilingual entity descriptions in MEP (ablation) | MEP · XNLI | test accuracy: drops by 0.1 | Helped |
| Use contrastive learning rather than relation classification with cross-entropy for Object Entailment (ablation) | OE · MLQA | F1: drops by 1.0 | Helped |
Closest papers by meaning
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.87)
- Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book? (ICLR 2025 · similarity 0.86)
- INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.86)
- XCOT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning (AAAI 2025 · similarity 0.86)
- XIFBench: Evaluating Large Language Models on Multilingual Instruction Following (NeurIPS 2025 · similarity 0.85)
- Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.