Research map / Large language models / Language, safety and interpretability
Large Language Models Are Read/Write Policy-Makers for Simultaneous Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LSG, which lets an off-the-shelf LLM act as its own policy-maker for simultaneous generation by comparing its next-token distributions under the current input and under a wait-1 baseline input (KL threshold plus confidence check and range constraint), with no policy training.
Problem
Simultaneous generation (translation, streaming ASR) needs a policy deciding when to write; existing methods use encoder-decoder models trained with complex dynamic programming, and prior LLM-based methods rely on external fixed or encoder-decoder policies.
Approach
Use the LLM's own next-token distributions: derive an enhanced policy from a wait-1 baseline by writing when the KL divergence between the distribution under current input and under the baseline input exceeds a threshold, or when the LLM's max probability exceeds a confidence threshold; bound the search range with a range constraint [L, U].
Key results
- On CoVoST2 Fr-En streaming ASR, LSG reaches WER 23.76 at AL 4342 ms versus Whisper-base at 38.04 WER and Wav2Vec2-large at 26.17 WER (AL 5684 ms).
- On WMT15 De-En, LSG with [7,6] reaches SacreBLEU 33.41 at AL 8.95 words, and the ablation shows removing the range constraint drops BLEU to 21.95 at AL 3.62.
- Claims state-of-the-art on SimulT2TT and SimulS2TT over encoder-decoder and prior LLM-based methods, with about 3 s computation-aware delay on Fr-En.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the confidence condition (ablation) | LSG · WMT15 De→En | SacreBLEU: 31.34 -> 31.60; 32.72 -> 33.22 | Helped |
| Add the range constraint (ablation) | LSG · WMT15 De→En | SacreBLEU: 21.95 -> 31.60; 29.90 -> 33.22 | Helped |
| Use a 320 ms segment size instead of 640 ms (ablation) | LSG · CoVoST2 Fr→En SimulS2TT | SacreBLEU: 31.71 vs 32.20; 36.08 vs 36.19 | Mixed |
| Increase the [L, U] range from [1, 4] to [7, 6] | LSG · WMT15 De→En | SacreBLEU: 28.60 -> 33.41 | Helped |
| Increase the [L, U] range from [1, 4] to [7, 6] | LSG · MuST-C En→De | SacreBLEU: 19.18 -> 29.87 | Helped |
| Increase the [L, U] range from [1, 4] to [5, 6] | LSG · CoVoST2 Fr→En SimulS2TT | SacreBLEU: 32.10 -> 37.76 | Helped |
| Use LSG instead of Wav2Vec2-large | CoVoST2 Fr→En streaming ASR · CoVoST2 Fr→En streaming ASR | WER: 26.17 -> 31.71 | Hurt |
| Use LSG instead of Whisper-base | CoVoST2 Fr→En streaming ASR · CoVoST2 Fr→En streaming ASR | WER: 38.04 -> 23.76 | Helped |
Surprising findings
- Removing the range constraint lowers BLEU substantially (21.95 vs 31.60 at a similar latency), showing outlier policies hurt quality.
Limitations
- Relies on a hand-chosen wait-1 baseline policy and tuned thresholds (delta, alpha) per task and direction.
- Speech LLM inference latency makes it slower than lightweight encoder-decoder systems in computation-aware terms.
Experimental setup
Datasets: WMT15 De-En MuST-C En-De CoVoST2 Fr-En WMT22 Zh-En
Benchmarks: WMT15 De-En newstest2015 MuST-C En-De CoVoST2 Fr-En WMT22 Zh-En
Metrics: Average Lagging SacreBLEU COMET WER computation-aware latency sufficiency rate
Baselines: wait-k MMA ITST HMT Agent-SiMT DiSeg StreamSpeech Wav2Vec2-large Whisper-base
Builds on: wait-k policy Agent-SiMT HMT SimulEval Average Lagging
Closest papers by meaning
- SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation (NeurIPS 2025 · similarity 0.88)
- Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.88)
- LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs (ICLR 2025 · similarity 0.86)
- Hansel: Output Length Controlling Framework for Large Language Models (AAAI 2025 · similarity 0.86)
- UniAudio: Towards Universal Audio Generation with Large Language Models (ICML 2024 · similarity 0.86)
- Divergence-Guided Simultaneous Speech Translation (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.