Research map / Architectures and efficiency / Attention and transformers
Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that transformer encoders can perform audio-to-text alignment internally via self-attention, enabling an Aligner-Encoder ASR model trained with frame-wise cross-entropy and a light text-only LSTM decoder that is faster than RNN-T and AED.
Problem
Standard end-to-end ASR (RNN-T, AED) leaves alignment to the decoder, requiring dynamic programming (RNN-T) or costly cross-attention (AED), which makes training and inference expensive.
Approach
The Conformer encoder learns to move information into label-aligned positions during its forward pass via self-attention, so the decoder can be a light, text-only LSTM that emits one token per frame trained with frame-wise cross-entropy, with no blanks, no dynamic programming and no cross-attention.
Key results
- On LibriSpeech, Aligner reaches 2.3% test-clean and 5.1% test-other WER versus RNN-T 2.1% and 4.6%, and AED 2.4% and 5.5%.
- Aligner total inference time is estimated at 2x faster than RNN-T and 16x faster than AED in a representative setting (T=300, U=100).
- Decoder+loss training step time drops from 290ms (RNN-T) to 29ms (Aligner), with 18% lower peak memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RNN-T decoding and loss with the Aligner-Encoder approach | Conformer ASR · Voice Search test sets | WER (%): 3.6 -> 3.7; 12.6 -> 12.5; 14.6 -> 13.1; 20.5 -> 20.4 | Mixed |
| Replace CTC with the Aligner-Encoder approach | Conformer ASR · LibriSpeech test-clean and test-other | WER (%): 2.8 -> 2.3; 6.4 -> 5.1 | Helped |
| Replace AED with the Aligner-Encoder approach | Conformer ASR · LibriSpeech test-clean and test-other | WER (%): 2.4 -> 2.3; 5.5 -> 5.1 | Helped |
| Use an independent non-autoregressive decoder instead of an autoregressive decoder (ablation) | Aligner · Voice Search test sets | WER (%): 4.3 -> 4.5; 14.3 -> 15.3; 17.8 -> 27.3; 23.8 -> 23.5 | Mixed |
| Increase the non-AR Aligner vocabulary size to 32K (ablation) | Non-AR Aligner · Voice Search | WER | Helped |
| Train with concatenated utterances up to the maximum test length (ablation) | Aligner · LibriSpeech test-clean All | WER (%): 4.8 -> 2.3 | Helped |
| Train with concatenated utterances up to the maximum test length (ablation) | Aligner · LibriSpeech test-other All | WER (%): 6.5 -> 5.1 | Helped |
| Chunk the long-form audio and reset the prediction network at chunk boundaries (ablation) | Aligner · YouTube long-form test | WER (%): 7.6 -> 7.5 | Helped |
| Prime the reset prediction network with 10 tokens of history (ablation) | Chunked Aligner · YouTube long-form test | WER (%): 7.5 -> 7.3 | Helped |
| Use Aligner instead of RNN-T for decoding | Conformer ASR · LibriSpeech inference (T=300, U=100) | Total inference time (ms): 108 -> 51 | Helped |
| Use Aligner instead of AED for decoding | Conformer ASR · LibriSpeech inference (T=300, U=100) | Total inference time (ms): 832 -> 51 | Helped |
| Use the Aligner training loss and decoder rather than RNN-T loss and decoder | LibriSpeech Conformer · LibriSpeech training step | Decoder+Loss time (ms): 290 -> 29 | Helped |
Surprising findings
- Alignment emerges abruptly within one or two self-attention layers (layers 14-15) of a Conformer encoder, rather than gradually.
- Aligners generalize poorly to utterances longer than the training length (test-clean >21s WER 28.0%) unless training includes concatenated long utterances; chunked inference reaches 7.3% on YouTube vs RNN-T's 6.8% unsegmented.
Limitations
- Degrades on utterances longer than the maximum training length; requires concatenated training data or chunking for long-form.
- Cannot downsample encoder frames below the label length; non-AR variant performs significantly worse than CTC on longer utterances.
- Experiments are limited to English ASR on proprietary and LibriSpeech data, with streaming and multilingual use left for future work.
Experimental setup
Datasets: LibriSpeech Voice Search (proprietary) YouTube long-form (proprietary)
Benchmarks: LibriSpeech test-clean LibriSpeech test-other Voice Search main and rare-word sets (Maps, News, Queries) YouTube long-form
Metrics: WER training step time inference time peak memory
Baselines: RNN-T AED CTC Non-AR Aligner Monotonic RNN-T Conformer AED (148M)
Builds on: RNN-Transducer attention-based encoder-decoder Conformer Rotary Position Embedding CTC
Closest papers by meaning
- Zipformer: A faster and better encoder for automatic speech recognition (ICLR 2024 · similarity 0.87)
- StableMask: Refining Causal Masking in Decoder-only Transformer (ICML 2024 · similarity 0.86)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.86)
- MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer (ICLR 2025 · similarity 0.85)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.85)
- How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.