Research map / Large language models / Language, safety and interpretability
PolyVoice: Language Models for Speech to Speech Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PolyVoice, a two-LM speech-to-speech translation framework that pairs a decoder-only unit-based translation LM (U-XLM) with a VALL-E X-style unit-to-speech audio LM (U-SLM) that preserves source speaker voice, using unsupervised HuBERT units to support unwritten languages.
Problem
Speech-to-speech translation has been dominated by encoder-decoder models and speaker-embedding or text-dependent synthesis, which struggle to preserve source speaker voice and to handle unwritten languages that lack transcripts.
Approach
Treat S2ST as language modeling over discrete HuBERT semantic units with a decoder-only cross-lingual model using prompt templates that unify ASR, MT, S2ST and TTS data, followed by a VALL-E X-style unit-based audio LM with SoundStream codec tokens and a duration LM to clone the source speaker's voice in the output.
Key results
- On Chinese-English EMIME, PolyVoice (S2UT+U2S) reaches 29.30 ASR-BLEU and 3.35 naturalness vs VALL-E X's 30.66 ASR-BLEU and 3.54 naturalness in the same table.
- On English-Spanish CVSS without any Spanish text, PolyVoice achieves 18.3 ASR-BLEU, 0.34 ASV and 3.60 naturalness.
- Decoder-only U-XLM improves ASR-BLEU from 16.8 (encoder-decoder) to 20.7 on the Chinese-English setup, and to 22.0 with U2S synthesis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PolyVoice instead of VALL-E X | EMIME Chinese→English S2ST · EMIME Chinese→English | Naturalness: 3.54 -> 4.10 | Helped |
| Use PolyVoice instead of VALL-E X | EMIME Chinese→English S2ST · EMIME Chinese→English | ASR-BLEU: 30.66 -> 29.40 | Hurt |
| Use oracle target text | VALL-E X on EMIME · EMIME Chinese→English | ASR-BLEU: 30.66 -> 86.78 | Helped |
| Use oracle target semantic units | PolyVoice on EMIME · EMIME Chinese→English | ASR-BLEU: 29.40 -> 76.10 | Helped |
| Use decoder-only rather than encoder-decoder architecture (ablation) | S2UT · EMIME Chinese→English | ASR-BLEU: 16.8 -> 20.7 | Helped |
| Add U2S (ablation) | Encoder-Decoder S2UT · EMIME Chinese→English | ASR-BLEU: 16.8 -> 18.7 | Helped |
| Add U2S (ablation) | Decoder-only S2UT · EMIME Chinese→English | ASR-BLEU: 20.7 -> 22.0 | Helped |
| Add multi-task training (ablation) | S2S model · EMIME Chinese→English | S2ST BLEU: 22.2 -> 29.4 | Helped |
| Add the semantic2dur duration model (ablation) | U2S · Speech resynthesis | WER: 31.93 -> 6.40 | Helped |
| Use mHuBERT_zh_en semantic units instead of the default units (ablation) | U2S · Speech resynthesis | WER: 6.40 -> 4.76 | Helped |
Surprising findings
- Predicted units yield higher naturalness than ground-truth-target conditioned output, while translation quality drops relative to VALL-E X.
- Removing the duration model makes WER increase dramatically (4.07 to 31.93), indicating duration is essential for unit-based synthesis.
Limitations
- Translation quality (ASR-BLEU) is lower than VALL-E X, with a large gap when oracle target text is used for synthesis.
- Relies on large in-house ASR, MT and TTS data that are not public, limiting reproducibility.
- Semantic units extracted without supervision introduce errors and retain some acoustic information.
Experimental setup
Datasets: LibriLight In-house Chinese ASR In-house Chinese-English MT GigaSpeech WenetSpeech EMIME CVSS
Benchmarks: EMIME (Chinese-English) CVSS (English-Spanish)
Metrics: ASR-BLEU ASV score naturalness (NISQA) WER CER
Baselines: VALL-E X Cascade (ASR+MT+YourTTS) Encoder-Decoder U-XLM U-XLM alone with unit vocoder
Builds on: VALL-E X VALL-E AudioLM HuBERT SoundStream
Closest papers by meaning
- TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation (NeurIPS 2024 · similarity 0.87)
- High-Fidelity Simultaneous Speech-To-Speech Translation (ICML 2025 · similarity 0.85)
- Divergence-Guided Simultaneous Speech Translation (AAAI 2024 · similarity 0.85)
- LAMA-UT: Language Agnostic Multilingual ASR Through Orthography Unification and Language-Specific Transliteration (AAAI 2025 · similarity 0.84)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.84)
- Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.