Research map / Large language models / Language, safety and interpretability
Multimodal and Multilingual Embeddings for Large-Scale Speech Mining
Paul-Ambroise Duquenne, Hongyu Gong, Holger Schwenk
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add mined speech-translation data to training | LNA bilingual S2T · CoVoST2 De-En | BLEU: 24.4 -> 26.4 | Improved |
| add mined speech-translation data to training | LNA bilingual S2T · CoVoST2 Es-En | BLEU: 29.9 -> 31.6 | Improved |
| add mined speech-translation data to training | LNA bilingual S2T · CoVoST2 Fr-En | BLEU: 30.7 -> 32.0 | Improved |
| add mined data to MuST-C training | S2T Transformer · MuST-C en-es | BLEU: 27.2 -> 28.7 | Improved |
| add mined data to MuST-C training | S2T Transformer · MuST-C en-fr | BLEU: 32.9 -> 34.4 | Improved |
| add mined data to MuST-C training | S2T Transformer · MuST-C en-ru | BLEU: 15.3 -> 16.1 | Improved |
| add mined data to MuST-C training | S2T Transformer · MuST-C en-vi | BLEU: 21.4 -> 21.8 | Improved |
| use separate per-language speech encoder instead of one multilingual encoder (ablation) | speech encoder · CoVoST2 ru audio vs en translations | error rate (%): 40.54 -> 6.86 | Improved |
| use separate per-language speech encoder instead of one multilingual encoder (ablation) | speech encoder · CoVoST2 de/fr/es audio vs en translations | error rate (%): fr 2.05 -> 1.86; de 3.36 -> 3.73; es 1.66 -> 1.78 | Mixed |
| train with both transcription and translation teachers instead of one (ablation) | multilingual speech encoder · CoVoST2 audio vs en translations (de, fr, es) | error rate (%): de 3.58/4.06 -> 3.36 | Improved |
| train with both teachers instead of one (ablation) | multilingual speech encoder · CoVoST2 ru audio vs transcriptions/translations | error rate (%): 25.63 -> 36.19 (transcriptions) | Hurt |
| increase speech encoder training data (ablation) | French speech encoder · CoVoST2 fr similarity search | error rate | Improved |
Closest papers by meaning
- Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (NeurIPS 2018 · similarity 0.89)
- MATS: An Audio Language Model under Text-only Supervision (ICML 2025 · similarity 0.87)
- EuroSpeech: A Multilingual Speech Corpus (NeurIPS 2025 · similarity 0.87)
- MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025 · similarity 0.86)
- WorldSpeech: A Multilingual Speech Corpus from Around the World (NeurIPS 2026 · similarity 0.86)
- Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
