Research map / Label-efficient and robust learning / Recognition and long-tail
REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASR
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reborn trains a CNN segmentation model with REINFORCE (rewards from phoneme LM perplexity plus edit-distance and length regularizers) alternating with wav2vec-U-style GAN training of the phoneme predictor, for unsupervised ASR.
Problem
Unsupervised ASR must map variable-length, unknown-boundary speech segments to phonemes without paired speech-text data; prior methods use handcrafted or separately learned segmentation that is suboptimal for the phoneme predictor.
Approach
Make the segmentation model trainable and tailored to the phoneme predictor: train it with policy gradient (REINFORCE) to favor segmentations whose de-duplicated phoneme outputs have lower LM perplexity, regularized by edit-distance and length rewards relative to the previous iteration, and iterate with GAN training of the phoneme predictor.
Key results
- On LibriSpeech 100h, Reborn reaches 8.9% PER / 13.1% WER on test-clean without self-training, and with HMM self-training 5.4 / 9.6, outperforming prior SoTA by about 5% PER and 6% WER.
- On TIMIT, Reborn achieves 12.4% PER with greedy decoding, beating EURO (WavLM) at 14.6% on core-test (no LM at decoding).
- On MLS, Reborn averages 32.0% WER across six languages vs 35.9% for wav2vec-U 2.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Reborn instead of wav2vec-U 2.0 + HMM self-training | LibriSpeech ASR · LibriSpeech test-clean | PER: 10.3 -> 8.9 | Helped |
| add HMM self-training | Reborn · LibriSpeech test-clean | PER: 8.9 -> 5.4 | Helped |
| apply Reborn instead of wav2vec-U 2.0 | unsupervised ASR · TIMIT core-test | PER: 19.8 -> 13.5 | Helped |
| apply Reborn instead of wav2vec-U 2.0 | unsupervised ASR · MLS, six-language average | WER: 35.9 -> 32.0 | Helped |
| apply Reborn instead of wav2vec-U 2.0 | unsupervised ASR · MLS Portuguese | WER: 48.5 -> 51.5 | Hurt |
| add the edit-distance reward (ablation) | Reborn reward design · LibriSpeech test-clean, stage 2 | PER: 12.9 -> 12.1 | Helped |
| add the length-difference reward to PPL and edit rewards (ablation) | Reborn reward design · LibriSpeech test-clean, stage 2 | PER: 12.1 -> 11.9 | Helped |
| add behavior-cloning initialization (ablation) | Reborn segmentation model · LibriSpeech test-clean, stage 2 | PER: 14.2 -> 11.9 | Helped |
| add boundary merging (ablation) | Reborn stage 2 GAN training · LibriSpeech test-clean | mean PER: 13.9 ± 2.1 -> 12.0 ± 0.1 | Helped |
| initialize GAN parameters from the previous iteration instead of randomly (ablation) | Reborn stage 2 GAN training · LibriSpeech test-clean | mean PER: 14.1 ± 0.4 -> 12.0 ± 0.1 | Helped |
| replace wav2vec-U segmentation with Reborn boundaries | the same phoneme prediction model · LibriSpeech test-clean | PER: 19.3 -> 12.9 | Helped |
| replace wav2vec 2.0 features with HuBERT features (ablation) | Reborn · LibriSpeech test-clean, iteration 1 stage 2 | PER: 11.9% -> 10.4% | Helped |
Surprising findings
- Reborn's learned segmentation produces more segments than phonemes (smaller than phone units), yielding lower PER despite lower boundary F1 than oracle and prior segmentation.
- Reborn with HMM self-training outperforms wav2vec-U trained with oracle boundaries on LibriSpeech.
Limitations
- Requires lexicon/phonemicization of unpaired text, so it does not work directly at word level.
- Not evaluated on low-resource languages or noisy environments.
- Poor initialization of the phoneme predictor limits gains; iterative training may amplify dataset biases.
Experimental setup
Datasets: LibriSpeech TIMIT Multilingual LibriSpeech
Benchmarks: LibriSpeech TIMIT Multilingual LibriSpeech
Metrics: PER WER boundary precision/recall/F1
Baselines: wav2vec-U wav2vec-U 2.0 EURO wav2vec-U 2.0 + HMM self-training EURO (WavLM)
Builds on: wav2vec-U wav2vec 2.0 REINFORCE EURO
Closest papers by meaning
- Unsupervised Speech Recognition via Segmental Empirical Output Distribution Matching (ICLR 2019 · similarity 0.88)
- Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models (NeurIPS 2024 · similarity 0.83)
- SyllableLM: Learning Coarse Semantic Units for Speech Language Models (ICLR 2025 · similarity 0.83)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.83)
- AURORA: Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation (AAAI 2026 · similarity 0.83)
- ASRU: Activation Steering Meets Reinforcement Unlearning for Multimodal Large Language Models (ICML 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.