Research map / Label-efficient and robust learning / Recognition and long-tail
High-Fidelity Simultaneous Speech-To-Speech Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Hibiki, a decoder-only multistream language model for simultaneous French-English speech-to-speech and speech-to-text translation with voice transfer, trained on weakly-supervised contextually aligned synthetic data.
Problem
Simultaneous speech-to-speech translation requires real-time decisions about when to emit translation, and paired chunk-level speech data is scarce; prior systems rely on complex inference policies that are hard to batch and often lack voice transfer and naturalness.
Approach
A decoder-only multistream RQ-Transformer jointly models source and target Mimi audio tokens plus an inner-monologue text stream; word-level delays are derived from the perplexity of an off-the-shelf MT model (MADLAD-3B) ("contextual alignment") and enforced via silence insertion or alignment-aware TTS, so inference is plain temperature sampling with classifier-free guidance on speaker similarity.
Key results
- On CVSS-C Fr-En short-form, Hibiki reaches 39.2 BLEU and 35.5 ASR-BLEU vs Seamless 37.0/33.8, with speaker similarity 0.41 vs 0.30.
- On long-form Audio-NTREX, Hibiki achieves 27.5 BLEU and 0.48 speaker similarity vs Seamless 25.4 and 0.43, at ~0.8s higher LAAL.
- Human MOS: Hibiki naturalness 3.73 vs Seamless 2.18 and ground-truth 4.12; quality 3.78 vs 2.22.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Hibiki instead of offline speech-translation baselines | French-English speech translation · French-English offline speech translation | ASR-BLEU: 28.5 -> 30.5 | Helped |
| Use Hibiki instead of Seamless | French-English simultaneous translation · CVSS-C Fr-En test short-form | ASR-BLEU: 37.0 -> 39.2 | Helped |
| Use Hibiki instead of Seamless | French-English simultaneous translation · CVSS-C Fr-En test short-form | LAAL: 2.8 -> 3.4 | Hurt |
| Use Hibiki instead of Seamless | French-English simultaneous translation · Audio-NTREX long-form | ASR-BLEU: 25.4 -> 27.5 | Helped |
| Use Hibiki instead of Seamless | French-English simultaneous translation · Audio-NTREX long-form | LAAL: 4.2 -> 5.0 | Hurt |
| Use Hibiki instead of Seamless | Speech translation human evaluation · Human evaluation | Naturalness MOS: 2.18 ± 0.09 -> 3.73 ± 0.09 | Helped |
| Add contextual alignment instead of using no lag (ablation) | Hibiki training data · French-English simultaneous translation | ASR-BLEU: 4.2 -> 26.6 | Helped |
| Add contextual alignment instead of using a constant 2-second lag (ablation) | Hibiki training data · French-English simultaneous translation | ASR-BLEU: 10.0 -> 26.6 | Helped |
| Replace contextual alignment with a constant 10-second lag (ablation) | Hibiki training data · French-English simultaneous translation | LAAL: 5.0 -> 9.02 | Hurt |
| Replace contextual alignment with sentence alignment (ablation) | Hibiki training data · French-English simultaneous translation | LAAL: 5.0 -> 21.49 | Hurt |
| Add the output text stream (Inner Monologue) (ablation) | Hibiki · French-English simultaneous translation | ASR-BLEU: 17.1 -> 26.6 | Helped |
| Add audio pretraining (ablation) | Hibiki · French-English simultaneous translation | ASR-BLEU: 14.6 -> 26.6 | Helped |
Surprising findings
- Hibiki reaches 47.9 ASR-BLEU against MADLAD text translations versus 35.5 against CVSS-C ground-truth, suggesting it mimics MADLAD outputs more than references.
- Classifier-free guidance with gamma=10 degrades ASR-BLEU from 26.6 to 18.9 despite higher speaker similarity trade-offs, indicating unintelligible speech at extreme guidance.
Limitations
- Evaluated on a single French-to-English translation task; scaling to more languages requires TTS training per language.
- Lower ASR-BLEU against ground truth (35.5) than against MADLAD references (47.9), indicating pseudo-target bias.
- Latency (LAAL 5.0 on long-form) is higher than Seamless (4.2) and end offset is slightly worse on short-form.
Experimental setup
Datasets: CVSS-C Fr-En CVSS-T VoxPopuli NTREX Common Crawl-based text corpus Wikipedia
Benchmarks: CVSS-C Fr-En test Audio-NTREX VoxPopuli European Parliament interpretations
Metrics: ASR-BLEU speaker similarity end offset LAAL MOS quality MOS naturalness
Baselines: Seamless StreamSpeech Translatotron 2 UnitY S2UT DASpeech
Builds on: Moshi RQ-Transformer Mimi MADLAD-3B CVSS
Closest papers by meaning
- Simultaneous Speech-to-Speech Translation Without Aligned Data (ICML 2026 · similarity 0.93)
- Divergence-Guided Simultaneous Speech Translation (AAAI 2024 · similarity 0.89)
- REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translation (AAAI 2026 · similarity 0.89)
- Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion (ICLR 2026 · similarity 0.87)
- SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation (NeurIPS 2025 · similarity 0.87)
- TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.