Research map / Label-efficient and robust learning / Recognition and long-tail
Align With Purpose: Optimize Desired Properties in CTC Models with a General Plug-and-Play Framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Align With Purpose (AWP), a plug-and-play hinge-loss term added to CTC that prioritizes alignments by a desired property, demonstrated on streaming latency reduction and minimum-WER training for ASR at up to 280K hours.
Problem
CTC treats all perfect alignments equally and all imperfect alignments equally, so properties such as token emission latency and WER cannot be controlled during training.
Approach
Sample N alignments from a CTC model, apply a property-specific function f_prop that produces an improved alignment, and add a hinge loss on the probability pairs to the CTC loss; no modification of the CTC forward-backward algorithm is needed.
Key results
- Online Stacked ResNet on LV-35K with AWP reduces drift latency from 341 ms to -251 ms (TL 771 to 179 ms), with WER rising from 2.72 to 3.28.
- AWP for mWER reduces relative WER by 2.2% on test-clean and 4% on test-other for Stacked ResNet, and 2.1% and 4.5% for Wav2Vec2.
- On LS-960, AWP gives DL -79 ms at WER 4.38, versus Peak First CTC 186 ms at 4.41 and Bayes Risk 63 ms at 4.78.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AWP for low-latency optimization | Stacked ResNet Online · Libri Test-Clean, LS-960 | DL (ms): 278 -> -79 | Helped |
| add AWP for low-latency optimization | Stacked ResNet Online · Libri Test-Clean, LV-35K | WER: 2.72 -> 3.28 | Hurt |
| add AWP for low-latency optimization | Stacked ResNet Online · Libri Test-Clean, Internal-280K | DL (ms): 249 -> 50 | Helped |
| add AWP for low-latency optimization | Conformer Online · Libri Test-Clean, LS-960 | DL (ms): 2 -> -172 | Helped |
| add AWP for minimum WER optimization | Stacked ResNet · Libri Test-Clean | WER: 2.63 -> 2.57 | Helped |
| add AWP for minimum WER optimization | Stacked ResNet · Libri Test-Other | WER: 7.46 -> 7.16 | Helped |
| add AWP for minimum WER optimization | Wav2Vec2 · Libri Test-Other | WER: 5.82 -> 5.56 | Helped |
| increase the AWP loss weight to 0.01 (ablation) | Stacked ResNet low-latency model · Libri Test-Clean, LS-960 | DL (ms): 278 -> -402 | Helped |
| increase the AWP loss weight to 0.01 (ablation) | Stacked ResNet low-latency model · Libri Test-Clean, LS-960 | WER: 4.06 -> 6.52 | Hurt |
| shift alignments by 2 tokens instead of 1 (ablation) | AWP low-latency property function · Libri Test-Clean, LS-960 | DL (ms): -79 -> -68 | Mixed |
| shift alignments by 4 tokens instead of 1 (ablation) | AWP low-latency property function · Libri Test-Clean, LS-960 | DL (ms): -79 -> 21 | Hurt |
| correct 2 words instead of 1 (ablation) | AWP minimum-WER property function · Libri Test-Clean | WER: 2.57 -> 2.54 | Helped |
Surprising findings
- Some AWP-trained online models reach negative drift latency, i.e. total latency below the lower bound set by the future context alone.
- Sampling with Gumbel-Softmax instead of standard sampling had negligible effect on results.
Limitations
- Requires careful design of the property function f_prop for each property.
- Only one property enhanced at a time; multi-property trade-offs not explored.
- Hyperparameters (alpha, lambda, start epoch) need tuning and drive the latency-WER trade-off.
Experimental setup
Datasets: LibriSpeech LibriVox subset (LV-35K) Internal-280K VoxPopuli (pre-training of Wav2Vec2)
Benchmarks: LibriSpeech test-clean LibriSpeech test-other
Metrics: WER drift latency (DL) total latency (TL)
Baselines: Peak First CTC TrimTail Bayes Risk CTC MWER_OPT Stacked ResNet offline/online Conformer online
Builds on: CTC MWER optimization Peak First CTC TrimTail Bayes Risk CTC
Closest papers by meaning
- CR-CTC: Consistency regularization on CTC for improved speech recognition (ICLR 2025 · similarity 0.91)
- Self-Distillation Regularized Connectionist Temporal Classification Loss for Text Recognition: A Simple Yet Effective Approach (AAAI 2024 · similarity 0.85)
- Test-Time Alignment for Large Language Models via Textual Model Predictive Control (ICLR 2026 · similarity 0.84)
- Uncertainty-Aware Self-Training for CTC-Based Automatic Speech Recognition (AAAI 2025 · similarity 0.84)
- Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.84)
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.