Research map / Architectures and efficiency / Attention and transformers
Overcoming Non-monotonicity in Transducer-based Streaming Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MonoAttn-Transducer, which adds learnable monotonic cross-attention to Transducer predictors and trains it via forward-backward posterior alignment, enabling non-monotonic streaming generation (simultaneous speech translation) without exponential state enumeration.
Problem
Transducer models decode input-synchronously, which limits their ability to handle non-monotonic alignments such as reordering in simultaneous translation; adding cross-attention to the predictor makes the state space exponential and breaks forward-backward training.
Approach
Use forward-backward posterior alignment between predictor states and encoder timesteps, estimated from a diagonal prior lattice, to compute expected monotonic context vectors via cumulative sums, so the predictor attends to source history with the same O(TU) complexity as standard Transducer, with chunk-synchronized alignment and a curriculum (offline pretrain then streaming finetune).
Key results
- On MuST-C En-De, MonoAttn-Transducer reaches 23.74 BLEU at LAAL 2305ms vs. Transducer 22.96 BLEU at 2312ms (chunk 1280ms), with offline 24.42 vs 23.10 BLEU.
- On En-Es, MonoAttn-Transducer-Posterior averages +0.75 BLEU and +0.95 COMET over Transducer across chunk sizes.
- On CVSS-C Fr-En speech-to-speech, at 320ms chunk MonoAttn-Transducer gets 18.3 ASR-BLEU vs 17.1 with LAAL 918ms vs 984ms for Transducer; training is 1.33x slower and peak memory 32GB vs 28GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add posterior-alignment monotonic attention to Transducer | Transducer · MuST-C En-Es across chunk sizes | BLEU: average improvement of 0.75 BLEU | Helped |
| Add posterior-alignment monotonic attention to Transducer | Transducer · MuST-C En-De across chunk sizes | COMET: average increase of as much as 2.06 COMET score | Helped |
| Add posterior-alignment monotonic attention to Transducer | Transducer · CVSS-C Fr-En, chunk 320 ms and offline | ASR-BLEU: 17.1 -> 18.3; 18.0 -> 19.3 | Helped |
| Replace posterior alignment inference with direct prior alignment (ablation) | MonoAttn-Transducer · MuST-C En-Es and En-De, chunk 320 ms | BLEU: average 1.03 BLEU lower than Transducer baseline | Hurt |
| Replace posterior alignment inference with direct prior alignment (ablation) | MonoAttn-Transducer · MuST-C En-Es and En-De, larger chunk sizes | BLEU | Mixed |
| Increase chunk size | MonoAttn-Transducer versus Transducer · MuST-C translation, especially latency above 1.5s and offline | BLEU: average improvement reaches 0.88 BLEU | Helped |
Surprising findings
- The prior-only variant (without posterior inference) underperforms Transducer by 1.03 BLEU at 320ms chunk size, while it matches or slightly exceeds the posterior variant at larger chunk sizes.
- TAED's gains over Transducer are larger at smaller chunk sizes, the opposite trend from MonoAttn-Transducer.
Limitations
- Training is 1.33x slower and uses more memory than the Transducer baseline.
- Requires a two-stage curriculum (offline pretraining then streaming finetuning) for stability.
- Prior-only training is unstable at small chunk sizes, so posterior inference is necessary.
Experimental setup
Datasets: MuST-C En-De MuST-C En-Es CVSS-C Fr-En
Benchmarks: MuST-C En-De MuST-C En-Es CVSS-C Fr-En
Metrics: BLEU COMET-22 ASR-BLEU AL LAAL DAL AP StartOffset
Baselines: Transducer CAAT TAED Wait-k RealTrans MU-ST EDAtt AlignAtt Seg2Seg NAST LSG
Builds on: Transducer monotonic attention CAAT TAED
Closest papers by meaning
- UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities (NeurIPS 2026 · similarity 0.85)
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation (NeurIPS 2025 · similarity 0.85)
- Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.84)
- Masked Audio Generation using a Single Non-Autoregressive Transformer (ICLR 2024 · similarity 0.83)
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.83)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.