Research map / Reinforcement learning / Visual and representation RL
Divergence-Guided Simultaneous Speech Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiG-SST, a simultaneous speech translation system that trains a prefix-enhanced translation model and a divergence-predicting read/write policy, combined with wait-k, evaluated on MuST-C En-De/Es/Fr.
Problem
Simultaneous speech translation needs both a translation model that handles partial speech input and a read/write policy that decides when to translate; prior work treats these separately, using fixed wait-k policies or heuristic adaptive policies and offline models for prefix translation.
Approach
Train the translation model with prefix-to-full and prefix-to-prefix ST samples, then train a lightweight policy module to predict the divergence between next-token distributions computed from partial vs complete input, writing when predicted divergence falls below a threshold, hybridized with wait-k for a maximum latency.
Key results
- On En-De, DiG-SST surpasses the closest competitors (DiSeg, MU-ST) by about 2 BLEU at around 1000ms latency and keeps a 3+ BLEU margin beyond 2000ms.
- Offline En-De BLEU of 26.9 vs 25.7 for ConST (beam 5), with average 31.8 across three directions.
- On En-Es, DiG-SST has a 4+ BLEU advantage over other models at around 1000ms latency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add prefix-enhanced training with prefix-to-full samples (ablation) | DiG-SST translation model · MuST-C En-De tst-COMMON | BLEU: >2 BLEU points around the 1000ms mark | Helped |
| Add offline MT loss (ablation) | DiG-SST translation model · MuST-C | translation quality | Helped |
| Add the divergence-based read/write policy module (ablation) | DiG-SST · MuST-C En-De tst-COMMON | BLEU | Helped |
| Use prefix-to-full rather than prefix-to-prefix sampling (ablation) | DiG-SST translation model · MuST-C En-De tst-COMMON | BLEU | Helped |
| Use prefix-to-full rather than prefix-to-prefix sampling (ablation) | DiG-SST translation model · MuST-C En-De tst-COMMON | hallucination rate | Helped |
| Add the divergence-based policy module (ablation) | DiG-SST prefix-enhanced model · MuST-C En-De tst-COMMON | hallucination rate | Helped |
| Combine predicted divergence scores with wait-k rather than using predicted divergence scores alone (ablation) | DiG-SST · MuST-C En-De tst-COMMON | BLEU | Helped |
| Raise the divergence threshold above 0.15 (ablation) | DiG-SST with wait-k · MuST-C En-De tst-COMMON | BLEU: λ exceeds 0.15 | Hurt |
| Use DiG-SST instead of ConST | MuST-C En-De tst-COMMON · MuST-C En-De tst-COMMON | BLEU: 25.7 -> 26.9 | Helped |
| Use DiG-SST instead of ConST | MuST-C En-Es tst-COMMON · MuST-C En-Es tst-COMMON | BLEU: 30.4 -> 30.9 | Helped |
| Use DiG-SST instead of ConST | MuST-C En-Fr tst-COMMON · MuST-C En-Fr tst-COMMON | BLEU: 36.8 -> 37.6 | Helped |
Surprising findings
- Using predicted divergence scores alone without wait-k yields markedly inferior quality, so the wait-k combination is necessary.
Limitations
- Relies on predicted divergence, which suffers from exposure bias since it is trained with reference translation history.
- Performance is sensitive to the threshold lambda; larger values cause premature writes and quality drops at higher latency.
- Pure divergence policy cannot be used without wait-k; comparisons are limited to MuST-C with some baselines missing for En-Fr.
Experimental setup
Datasets: MuST-C v1 (En-De, En-Es, En-Fr) LibriSpeech-960
Benchmarks: MuST-C tst-COMMON En-De MuST-C tst-COMMON En-Es MuST-C tst-COMMON En-Fr
Metrics: BLEU average lagging hallucination rate
Baselines: MU-ST MoSST RealTranS ITST DiSeg MMA-SLM SpeechT5 XSNET STEMM ConST
Builds on: wait-k policy wav2vec 2.0 SimulEval STEMM
Closest papers by meaning
- SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation (NeurIPS 2025 · similarity 0.91)
- High-Fidelity Simultaneous Speech-To-Speech Translation (ICML 2025 · similarity 0.89)
- TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation (NeurIPS 2024 · similarity 0.87)
- SimulPL: Aligning Human Preferences in Simultaneous Machine Translation (ICLR 2025 · similarity 0.86)
- Large Language Models Are Read/Write Policy-Makers for Simultaneous Generation (AAAI 2025 · similarity 0.86)
- Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.