Research map / Large language models / Language, safety and interpretability
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use XLM-R Large rather than mBERT for zero-shot transfer | Multilingual encoder evaluated on XTREME · XTREME tasks | average score: 59.8 -> 68.2 | Helped |
| Translate English training data into target languages rather than use zero-shot transfer | mBERT · XNLI | accuracy: 65.4 -> 74.6 | Helped |
| Translate English training data into target languages rather than use zero-shot transfer | mBERT · PAWS-X | accuracy: 81.9 -> 86.3 | Helped |
| Translate English training data into target languages rather than use zero-shot transfer | mBERT · XQuAD | F1: 64.5 -> 70.0 | Helped |
| Train on translated data from all languages jointly rather than separately | Translate-train mBERT · XNLI | accuracy: 74.6 -> 75.1 | Helped |
| Translate target-language test data into English for evaluation rather than use zero-shot transfer | BERT-large · TyDiQA-GoldP | F1: 59.7 -> 72.1 | Helped |
| Use the middle-layer average word embeddings rather than final-layer CLS embeddings (ablation) | mBERT sentence representations · BUCC | F1: CLS 3.88; Layer 6 51.29; Layer 7 62.51; Layer 8 64.32 | Helped |
| Use middle-layer average word embeddings rather than bottom-layer average word embeddings (ablation) | mBERT sentence representations · BUCC | F1: Layer 1-4 6.98; Layer 5-8 63.12 | Helped |
| Use middle-layer average word embeddings rather than top-layer average word embeddings (ablation) | mBERT sentence representations · BUCC | F1: Layer 5-8 63.12; Layer 9-12 53.97 | Helped |
Closest papers by meaning
- Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework (ICLR 2020 · similarity 0.91)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.90)
- INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.89)
- MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025 · similarity 0.89)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.89)
- The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.