Research map / Vision-language and multimodal / CLIP and zero-shot
A Comparison of Architectures and Pretraining Methods for Contextualized Multilingual Word Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LASER-top contextualized embeddings instead of MUSE static embeddings | NER tagging · English → Dutch zero-shot transfer | F1: 0.477 -> 0.578 | Helped |
| Use LASER-top instead of LASER-elmo's weighted combination of encoder layers (ablation) | NER tagging · English → German zero-shot transfer | F1: 0.414 -> 0.617 | Helped |
| Add multilingual LASER pretraining rather than use a randomly initialized two-layer LSTM (ablation) | LASER-based contextualized embeddings · NER, English → German zero-shot transfer | F1: 0.235 -> 0.617 | Helped |
| Use multilingual BERT rather than XLM's TLM pretraining | Zero-shot cross-lingual tagging · NER, English → German | F1: 0.579 -> 0.589 | No clear effect |
| Use multilingual BERT rather than XLM | Zero-shot cross-lingual tagging · NER, English → Basque | F1: 0.0811 -> 0.403 | Helped |
| Use LASER-top rather than BERT | Zero-shot cross-lingual tagging · NER, English → German | F1: 0.589 -> 0.617 | Helped |
| Use LASER-top rather than BERT | Zero-shot cross-lingual tagging · POS, English → Dutch and German | F1: Dutch: 0.658 -> 0.726; German: 0.722 -> 0.781 | Mixed |
| Apply joint training to LASER-top | LASER-top · NER, joint training scenario B versus scenario A | F1: English +0.012; Dutch +0.03; German +0.03; Spanish +0.047; Basque -0.004; Hungarian +0.028 | Mixed |
| Use joint training scenario B rather than scenario A | BERT · NER, joint training | F1: English -0.038; Dutch +0.017; German +0.012; Spanish +0.005; Basque -0.; Hungarian +0.057 | Mixed |
| Use joint training scenario B rather than scenario A | XLM · NER, joint training | F1: English +0.001; Dutch +0.006; German +0.022; Spanish +0.024; Basque -0.013; Hungarian -0.024 | Mixed |
| Use joint training scenario B rather than scenario A | LASER-top · POS, joint training | F1: English +0.004; Dutch +0.006; German +0.005; Spanish +0.004 | Mixed |
Closest papers by meaning
- Unsupervised Interlingual Semantic Representations from Sentence Embeddings for Zero-Shot Cross-Lingual Transfer (AAAI 2020 · similarity 0.90)
- Multilingual Alignment of Contextual Word Representations (ICLR 2020 · similarity 0.90)
- Minimally-Constrained Multilingual Embeddings via Artificial Code-Switching (AAAI 2016 · similarity 0.89)
- Normalization of Language Embeddings for Cross-Lingual Alignment (ICLR 2022 · similarity 0.88)
- MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025 · similarity 0.88)
- The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.