Research map / Large language models / Language, safety and interpretability
Machine-Created Universal Language for Cross-Lingual Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MUL, a machine-created universal intermediate language of discrete symbols built via multilingual MLM, inter-sentence contrastive learning, and VQ-CA, and uses it as input for cross-lingual transfer, reporting gains over XLM-R Base on XNLI, NER, MLQA, and Tatoeba.
Problem
Cross-lingual transfer has two approaches: multilingual pre-training (strong performance but implicit alignment) and translate-test (interpretable but weaker and unsuitable for word-level tasks because translation alters word order).
Approach
Create a machine-learned universal vocabulary of discrete symbols with an NL-to-MUL translator that maps shared concepts to the same symbol while preserving word order and language-specific words; training uses inter-sentence contrastive learning and vector quantization with cross-lingual alignment (VQ-CA).
Key results
- MUL Base reaches 77.6 average accuracy on XNLI versus 75.7 for XLM-R Base and 76.5 for InfoXLM, with the same ~277M parameters.
- MUL Base scores 63.0 on NER, 69.4/50.8 on MLQA, and 79.3 on Tatoeba, the best of the compared models in the paper.
- MUL Small (132M parameters, about half of XLM-R Base) matches XLM-R Base on XNLI (75.7 avg) with a 60K universal vocabulary.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Translate inputs into MUL using the parameter-matched MUL Base configuration | MUL Base vs. InfoXLM · XNLI (avg) | accuracy: 76.5 -> 77.6 | Helped |
| Translate inputs into MUL using the parameter-matched MUL Base configuration | MUL Base vs. XLM-R Base · NER | score: 61.9 -> 63.0 | Helped |
| Translate inputs into MUL using the parameter-matched MUL Base configuration | MUL Base vs. InfoXLM · MLQA | score (two reported values): 68.1 / 49.7 -> 69.4 / 50.8 | Helped |
| Translate inputs into MUL using the parameter-matched MUL Base configuration | MUL Base vs. InfoXLM · Tatoeba | score: 77.8 -> 79.3 | Helped |
| Add VQ-CA (ablation) | MUL · Word alignment (de-en, fr-en, zh-en average) | recall: 42.3 -> 51.2 | Helped |
| Add contrastive loss and VQ-CA (ablation) | MUL · Word alignment (de-en, fr-en, zh-en average) | recall: 11.5 -> 51.2 | Helped |
| Add contrastive loss and VQ-CA (ablation) | MUL · XNLI | score: 69.7 -> 74.0 | Helped |
| Use inter-sentence contrastive learning on 4 sentence pairs instead of 1 (ablation) | MUL · XNLI | score: 72.9 -> 74.0 | Helped |
| Use inter-sentence contrastive learning on 4 sentence pairs instead of 2 (ablation) | MUL · XNLI | score: 73.4 -> 74.0 | Helped |
Surprising findings
- Translating to a learned 60K-symbol universal vocabulary outperforms multilingual pre-training baselines even though the vocabulary is much smaller than the 250K multilingual vocabulary.
- Removing VQ-CA and contrastive loss together collapses word alignment recall to 11.5% and drops XNLI to 69.7.
Limitations
- Ablations skip MUL pre-training and only fine-tune, due to limited compute.
- Disambiguation is imperfect because concept distributions in the data are unbalanced (e.g., 'club' maps to one universal word in most cases).
- Pre-training used fewer steps than XLM-R Base due to resource limits.
Experimental setup
Datasets: CC-Net OPUS-100 XNLI CoNLL NER MLQA Tatoeba CoarseWSD-20
Benchmarks: XNLI NER MLQA Tatoeba
Metrics: accuracy F1 F1 (MLQA answer F1 / exact match) accuracy (Tatoeba retrieval) precision recall alignment error rate
Baselines: mBERT XLM XLM-R Base mT5 Base Unicoder InfoXLM
Builds on: VQ-VAE XLM-R mBERT InfoXLM translate-test
Closest papers by meaning
- Cross-Lingual Pre-Training Based Transfer for Zero-Shot Neural Machine Translation (AAAI 2020 · similarity 0.85)
- Cross-lingual Language Model Pretraining (NeurIPS 2019 · similarity 0.85)
- Cross-Lingual Natural Language Generation via Pre-Training (AAAI 2020 · similarity 0.84)
- On Learning Universal Representations Across Languages (ICLR 2021 · similarity 0.84)
- Mufu: Multilingual Fused Learning for Low-Resource Translation with LLM (ICLR 2025 · similarity 0.83)
- Constrained Decoding for Cross-lingual Label Projection (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.