Research map / Large language models / Language, safety and interpretability
LAMA-UT: Language Agnostic Multilingual ASR Through Orthography Unification and Language-Specific Transliteration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAMA-UT, a two-stage multilingual ASR pipeline that fine-tunes wav2vec2.0-XLS-R to emit Romanized character transcriptions and then uses a frozen LLM (GPT-4o-mini, LLaMA) to transliterate them into language-specific scripts.
Problem
Multilingual ASR that performs equitably across many languages without language-specific modules (adapters, heads, LMs), especially for low-resource and unseen languages, while using little training data.
Approach
Unify orthographies into Romanized character transcriptions via a universal CTC acoustic model, then treat conversion back to native orthography as a text-to-text transliteration task solved by a frozen LLM prompted with few-shot examples, with no language-specific modules.
Key results
- On 82 FLEURS languages covered by Whisper, MMS and the method, LAMA-UT achieves average CER 14.8% vs Whisper's 23.9% (about 60% relative reduction) and matches MMS-1162 (7.8% CER) only approximately; WER is 33.2% vs 42.9% for Whisper.
- On 25 unseen CommonVoice languages, LAMA-UT reaches 34.7% CER vs ASR-2K's 65.5%, and is comparable to MMS-ZS (29.2%) which uses lexicons and n-gram LMs.
- Romanization outperforms IPA as the intermediate representation (universal transcription generator CER 11.3 vs 14.4 on seen languages).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LAMA-UT instead of Whisper-large-v3 | Multilingual ASR on FLEURS · FLEURS (82-language average) | CER: 23.9 -> 14.8 | Helped |
| Use LAMA-UT instead of Whisper-large-v3 | Multilingual ASR on FLEURS · FLEURS (82-language average) | WER: 42.9 -> 33.2 | Helped |
| Use LAMA-UT instead of MMS-1162 | Multilingual ASR on FLEURS · FLEURS (82-language average) | CER: 7.8 -> 14.8 | Hurt |
| Use LAMA-UT instead of MMS-1162 | Multilingual ASR on FLEURS · FLEURS (82-language average) | WER: 28.8 -> 33.2 | Hurt |
| Use LAMA-UT instead of ASR-2K | Zero-shot ASR · CommonVoice 17.0 (25 unseen languages) | CER: 65.5 -> 34.7 | Helped |
| Use LAMA-UT instead of MMS-ZS + n-gram LM | Zero-shot ASR · CommonVoice 17.0 (25 unseen languages) | CER: 25.2 -> 34.7 | Hurt |
| Use Romanization instead of IPA for orthography unification (ablation) | Universal transcription generator · FLEURS seen languages | CER: 14.4 -> 11.3 | Helped |
| Use Romanization instead of IPA for orthography unification (ablation) | Universal transcription generator · CommonVoice unseen languages | CER: 35.1 -> 32.3 | Helped |
| Use few-shot prompting (5 examples) instead of zero-shot prompting (ablation) | GPT-4o-mini universal converter · FLEURS seen languages | CER: 16.6 -> 15.3 | Helped |
| Use few-shot prompting (5 examples) instead of zero-shot prompting (ablation) | GPT-4o-mini universal converter · FLEURS seen languages | WER: 39.3 -> 37.2 | Helped |
| Add zero-shot chain-of-thought prompting (ablation) | GPT-4o-mini universal converter · FLEURS seen languages | CER: 16.6 -> 18.2 | Hurt |
| Use few-shot plus zero-shot CoT prompting instead of zero-shot prompting (ablation) | GPT-4o-mini universal converter · FLEURS seen languages | CER: 16.6 -> 15.7 | Helped |
Surprising findings
- Converting predicted IPA transcriptions with few-shot examples sometimes outperformed passing ground-truth IPA without examples, attributed to LLMs rarely seeing IPA tokens.
- The largest end-to-end degradation after the LLM converter occurred for non-Latin, non-spaced scripts such as Thai, Khmer, Lao, Burmese and Cantonese.
Limitations
- Language-specific transliteration relies on a frozen LLM, causing large error propagation for non-Latin, unspaced scripts (e.g., Thai, Khmer, Lao).
- Average WER remains high (33.2%) and degrades on some languages, and results depend on a closed-source GPT-4o-mini for the best performance.
- Relies on uroman and Phonemizer for preprocessing and reference Romanization, with an unclear quality for Japanese and Chinese.
Experimental setup
Datasets: FLEURS CommonVoice 17.0
Benchmarks: FLEURS CommonVoice 17.0
Metrics: CER WER PER repetition rate
Baselines: Whisper-large-v3 MMS-1162 wav2vec2.0-phoneme + n-gram LM ASR-2K MMS-ZS MMS-ZS + n-gram LM
Builds on: wav2vec2.0-XLS-R MMS Whisper ASR-2K uroman
Closest papers by meaning
- WorldSpeech: A Multilingual Speech Corpus from Around the World (NeurIPS 2026 · similarity 0.89)
- SwitchLingua: The First Large-Scale Multilingual and Multi-Ethnic Code-Switching Dataset (NeurIPS 2025 · similarity 0.87)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.87)
- WAXAL: A Large-Scale Multilingual African Language Speech Corpus (NeurIPS 2026 · similarity 0.86)
- “Allot?” is “A Lot!” Towards Developing More Generalized Speech Recognition System for Accessible Communication (AAAI 2024 · similarity 0.86)
- Objective Soups: Multilingual Multi-Task Modeling for Speech Processing (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.