Research map / Label-efficient and robust learning / Knowledge distillation
Heuristic-free Knowledge Distillation for Streaming ASR via Multi-modal Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Heuristic-free KD, a self-distillation method for streaming ASR where a streaming model with cross-attention fusion of full-context target text acts as teacher, removing the need for an offline teacher and heuristic time shift τ.
Problem
Knowledge distillation for streaming ASR relies on a non-streaming teacher whose outputs must be heuristically shifted by τ frames to align with the streaming student, which is sub-optimal and requires tuning.
Approach
Self-distillation within a single streaming architecture: the student with an added cross-attention fusion of target-text embeddings serves as the teacher, sharing encoder and decoder parameters, so frame-level alignment matches and no τ shift or extra teacher is needed.
Key results
- With a 1040 ms look-ahead CTC student, Heuristic-free KD reaches 6.24% test-clean WER (13.94% RERR) vs. 7.42% baseline, beating SKD (6.97%) and Guided CTC (7.51%) with tau tuned.
- With a 480 ms look-ahead, Heuristic-free KD achieves 11.27% RERR on test-clean and 7.85% on test-other, while SKD and Guided CTC with tau=5 sometimes degrade the student.
- On hybrid Transducer-CTC (1040 ms), Heuristic-free KD yields 7.05% RERR on test-clean and 5.93% on test-other, outperforming SKD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use streaming self-distillation with full-context textual input instead of conventional offline-teacher KD | Streaming CTC FastConformer, 1040 ms look-ahead · LibriSpeech test-clean | WER: 6.78 % -> 6.24 % | Helped |
| Use streaming self-distillation with full-context textual input instead of conventional offline-teacher KD | Streaming CTC FastConformer, 480 ms look-ahead · LibriSpeech test-clean | WER: 5.52 % -> 4.96 % | Helped |
| Use streaming self-distillation with full-context textual input instead of streaming-to-streaming SKD | Hybrid Transducer-CTC FastConformer, 1040 ms look-ahead · LibriSpeech test-clean | WER: 5.39 % -> 5.14 % | Helped |
| Set KD loss weight λ to 0.250 (ablation) | Heuristic-free KD · LibriSpeech | WER | Helped |
Surprising findings
- Naive frame-to-frame KD without time shift (tau=0) makes the distilled student worse than its baseline for conventional methods.
- The proposed teacher does not collapse to copying the target text despite receiving it as input, attributed to the many-to-one CTC alignment.
Limitations
- Heuristic-free KD's teacher WER is not zero and it is compared against NeMo pretrained teachers rather than retrained ones, so comparisons rely on released checkpoints.
- Experiments limited to LibriSpeech (Common Voice Spanish only in appendix) and two architectures; Dual-mode ASR not directly compared due to architectural changes.
Experimental setup
Datasets: LibriSpeech train-clean-100 LibriSpeech train-clean-360 LibriSpeech train-other-500
Benchmarks: LibriSpeech dev-clean LibriSpeech dev-other LibriSpeech test-clean LibriSpeech test-other
Metrics: word error rate relative error rate reduction
Baselines: Guided CTC SKD Offline FastConformer-CTC teacher with time shift tau Dual-mode ASR (reported comparison only)
Builds on: SKD Guided CTC Dual-mode ASR FastConformer
Closest papers by meaning
- TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.86)
- LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale (CVPR 2025 · similarity 0.86)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.86)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.85)
- Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models (NeurIPS 2024 · similarity 0.85)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.