Research map / Label-efficient and robust learning / Recognition and long-tail
TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TransVIP, a cascade-in-one-model speech-to-speech translation system that jointly generates target text and coarse codec tokens, plus separate acoustic and isochrony encoders to preserve voice and duration, evaluated on French-English CVSS-T.
Problem
End-to-end speech-to-speech translation lags cascaded systems, is data-scarce, and typically fails to preserve speaker voice and output duration (isochrony) needed for video dubbing.
Approach
Decompose S2ST into a joint autoregressive translation model that emits text then first-layer codec tokens (optimizing P(Y',T|X)), conditioned on separate acoustic (speaker prompt embedding) and isochrony (position and VAD embeddings) encoders, followed by a textless non-autoregressive model that fills in remaining codec layers; also introduces SASCodec (semantically distilled codec) and Layer Bea
Key results
- Fr-En ASR-BLEU 32.60 vs 30.85 for SeamlessExpressive and 28.95 for SeamlessM4T Medium; speaker SIM 0.320 vs 0.256.
- SLC_0.2 in Fr-En 0.70 vs 0.50 for SeamlessExpressive, showing stronger isochrony control.
- Human CMOS naturalness preference for TransVIP over SeamlessExpressive higher by 0.46 (12 raters, 120 pairs).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TransVIP instead of SeamlessExpressive | Speech-to-speech translation · CVSS-T Fr-En test | Speech BLEU: 30.85 -> 32.60 | Helped |
| Use TransVIP instead of SeamlessExpressive | Speech-to-speech translation · CVSS-T Fr-En test | SIM: 0.256 -> 0.320 | Helped |
| Use TransVIP instead of SeamlessExpressive | Speech-to-speech translation · CVSS-T Fr-En test | AutoPCP: 2.74 -> 2.49 | Hurt |
| Use TransVIP instead of SeamlessExpressive | Speech-to-speech translation · CVSS-T En-Fr test | Speech BLEU: 27.39 -> 27.28 | Hurt |
| Use TransVIP instead of SeamlessExpressive | Speech-to-speech translation · CVSS-T En-Fr test | Text BLEU: 34.89 -> 33.02 | Hurt |
| Remove the acoustic embedding (ablation) | TransVIP joint translation model · CVSS-T Fr-En test | SIM: 0.320 -> 0.280 | Hurt |
| Remove the acoustic embedding (ablation) | TransVIP joint translation model · CVSS-T En-Fr test | SIM: 0.395 -> 0.362 | Hurt |
| Replace encoder isochrony control with no isochrony control (ablation) | TransVIP · Isochrony control ablation | SLC 0.2: 0.82 -> 0.63 | Hurt |
| Replace encoder isochrony control with decoder isochrony control (ablation) | TransVIP · Isochrony control ablation | SLC 0.2: 0.82 -> 0.75 | Hurt |
| Replace encoder isochrony control with decoder isochrony control plus future pause information (ablation) | TransVIP · Isochrony control ablation | SLC 0.2: 0.82 -> 0.77 | Hurt |
| Use greedy decoding instead of Layer Beam Search (ablation) | Textless NAR acoustic model · Fr-En | ASR-BLEU: 32.60 -> 32.30 | Hurt |
| Use text transcriptions as NAR model input instead of textless input (ablation) | NAR acoustic model · Fr-En | ASR-BLEU: 32.60 -> 31.52 | Hurt |
Surprising findings
- The textless NAR acoustic model matched or outperformed a text-conditioned version, attributed to semantic distillation in the codec.
- Using DAC codec caused the joint translation model to fail to generate reasonable output despite apparently converged training loss.
Limitations
- Trained only on French-English; limited data (about 5k hours) compared with over 50k hours in prior work.
- Occasionally alters tone, e.g., turning interrogatives into declaratives.
- Text BLEU lags SeamlessExpressive in En-Fr and Fr-En, attributed to smaller S2TT data and model size.
Experimental setup
Datasets: CVSS-T SeamlessAlign Common Voice v15 LibriLight VoxPopuli LibriHeavy SpeechMatrix
Benchmarks: CVSS-T fr-en test subset (300 utterances) LibriSpeech test-clean
Metrics: BLEU ASR-BLEU SIM AutoPCP Rate Pause SLC NISQA WER CMOS
Baselines: Textless S2UT XM SeamlessM4T Medium SeamlessExpressive ST + StyleTTS ST + ValleX
Builds on: SpeechTokenizer VALL-E DAC SeamlessM4T SpeechMatrix
Closest papers by meaning
- Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (ICML 2022 · similarity 0.90)
- Divergence-Guided Simultaneous Speech Translation (AAAI 2024 · similarity 0.87)
- PolyVoice: Language Models for Speech to Speech Translation (ICLR 2024 · similarity 0.87)
- High-Fidelity Simultaneous Speech-To-Speech Translation (ICML 2025 · similarity 0.86)
- DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation (NeurIPS 2024 · similarity 0.85)
- SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.