Research map / Label-efficient and robust learning / Recognition and long-tail
An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a self-learning ASR framework where a context-aware teacher (audio/text dialog context plus contrastive PF-CLC loss with a new online hard-negative mining method, Ohm) is distilled into a context-free streaming student model.
Problem
Single-utterance ASR in voice assistants fails to use conversational context and implicit user feedback (reformulations), and existing context-aware methods add run-time latency, storage, or labeling cost.
Approach
Train a non-streaming context-aware transducer teacher with explicit audio/text dialog context and past/future contrastive learning (CLC) using Ohm, an online clustering-based hard-negative batch construction that works within small per-GPU batch sizes; then distill to a context-free streaming student that needs no extra run-time compute.
Key results
- Teacher (200M) with FC audio context, learned text context, CLC+Ohm reaches 6.91% WERR on ALL and 9.58% on REF over baseline; 1B teacher reaches 9.24% on REF.
- On OD3 (200M), B/CX/C/O achieves 8.73% WER vs 11.90% baseline overall (26.2% WERR) and 30.8% WERR on repeat/rephrase subset.
- Distilled 1B student recovers up to 3.06% WERR on REF (31.94% distillation efficiency) without run-time context.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add feature concatenation audio context to the baseline | 200M teacher · Real-world ref | WERR: 7.04 | Helped |
| Use learned text embeddings instead of BERT text embeddings (ablation) | 200M teacher with text context · Real-world ref | WERR: 2.66 -> 5.66 | Helped |
| Combine feature concatenation audio with learned text context (ablation) | 200M teacher · Real-world ref | WERR: 7.04 -> 8.08 | Helped |
| Add future audio and text context to past-only audio and text context (ablation) | 200M teacher · Real-world ref | WERR: 5.08 -> 8.08 | Helped |
| Add future audio and text context to past-only audio and text context (ablation) | 200M teacher · Real-world all | WERR: 2.90 -> 4.70 | Helped |
| Add explicit context to the OD3 baseline (ablation) | 200M teacher · OD3 overall | WER: 11.90 -> 11.13 | Helped |
| Add CLC loss to explicit context (ablation) | 200M teacher · OD3 overall | WER: 11.13 -> 8.99 | Helped |
| Add Ohm to explicit context and CLC (ablation) | 200M teacher · OD3 overall | WER: 8.99 -> 8.73 | Helped |
| Add Ohm to context and CLC (ablation) | 200M teacher · Real-world ref | WERR: 8.68 -> 10.60 | Helped |
| Oversample reformulations at a 1:5 rate (ablation) | Teacher training · Real-world all | WERR: 0.46 | Helped |
| Distill the context-trained teacher into the single-utterance student | 1B student · Student distillation, all | WERR: 1.76 | Helped |
| Add CLC to context training (ablation) | 200M teacher · Real-world tail, domain-normalized all | WERR: 3.75 | Helped |
Surprising findings
- The 200M context-aware teacher outperforms a 1B baseline without context losses, showing context gains can exceed model-size gains.
- Adding CLC alone improves deletion rate substantially but hurts substitution and insertion; Ohm then improves substitution at the cost of deletions.
Limitations
- Evaluation relies on a proprietary real-world dataset that cannot be released, and OD3 is semi-synthetic.
- Context-heavy teacher degrades on home automation tasks (-22.68% WERR), suggesting over-reliance on context.
- Student recovers only a fraction (about 33%) of teacher gains without context.
Experimental setup
Datasets: PRETRAIN (200k+ hours de-identified ASR) closed-source dialogues (10M for teacher training) SSRD (semi-supervised single utterance reformulation dataset, ~25k hours) OD3
Benchmarks: OD3 closed-source ALL dataset closed-source REF (reformulation) dataset
Metrics: WER WERR BERT-S SERR distillation efficiency
Baselines: Whisper S Whisper L Conformer (Librispeech) Wav2Vec 2 Streaming Conformer CLC Baseline transducer without context
Builds on: CLC (contrastive learning from dialog context) Conformer transducer BIRCH clustering reservoir sampling
Closest papers by meaning
- Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics (ICLR 2025 · similarity 0.87)
- TA&AT: Enhancing Task-Oriented Dialog with Turn-Level Auxiliary Tasks and Action-Tree Based Scheduled Sampling (AAAI 2024 · similarity 0.87)
- Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training (ICLR 2025 · similarity 0.87)
- Aligning Spoken Dialogue Models from User Interactions (ICML 2025 · similarity 0.86)
- Language Model Can Listen While Speaking (AAAI 2025 · similarity 0.85)
- Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.