Research map / Image generation / Autoregressive and latent generation
UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniWav, an encoder-decoder speech foundation model that jointly pre-trains a masked-prediction (DinoSR-style self-distillation) encoder and a Flow Matching decoder from scratch on LibriLight, then evaluates it on ASR, in-context TTS, and tokenization.
Problem
Speech representation learning (discriminative) and generative modeling use separate foundation models with conflicting requirements, creating overhead; the paper asks whether one pre-trained model can serve both.
Approach
Jointly train a masked-audio-modeling representation encoder (self-distillation with online clustering from an EMA teacher) and a Flow Matching decoder conditioned on a learned weighted sum of encoder layers, end-to-end from scratch, so one encoder supports recognition, synthesis, and low-bitrate tokenization via a discrete bottleneck.
Key results
- On LibriSpeech test-other, UniWav (60k pre-training) reaches 4.0 WER with 960h fine-tuning, about 0.5% behind the best self-supervised baselines (data2vec 2.0 3.5).
- In-context TTS: UniWav reaches 2.5 ASR-WER and 0.635 speaker similarity, comparable to SpeechFlow (2.4 / 0.629) and better than VALL-E (3.8 / 0.508).
- At 1k bps tokenization, UniWav gets 5.6 WER and 3.72 UTMOS vs SpeechTokenizer's 9.1 WER and 2.08 UTMOS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add joint generative pre-training (ablation) | UniWav · LibriSpeech fine-tuned recognition, 100hr | WER: 17.5 -> 4.8 | Helped |
| add pre-training (ablation) | UniWav · LibriSpeech in-context TTS | speaker similarity: 0.532 -> 0.635 | Helped |
| add pre-training (ablation) | UniWav · LibriSpeech in-context TTS | ASR-WER: 2.6 -> 2.5 | No clear effect |
| include decoder and generative objective with encoder depth 24 (ablation) | UniWav · LibriSpeech dev-other, 200k steps | WER: 5.8 -> 5.5 | Helped |
| include decoder objective with encoder depth 12 (ablation) | UniWav · LibriSpeech dev-other, 200k steps | WER: 7.6 -> 8.1 | Hurt |
| increase encoder depth from 12 to 24 with decoder depth 12 (ablation) | UniWav · LibriSpeech in-context TTS, 200k steps | ASR-WER: 2.5 -> 2.5 | No clear effect |
| replace EnCodec latent with mel spectrogram input (ablation) | UniWav · LibriSpeech recognition, 200k steps | WER: 5.5 -> 5.6 | No clear effect |
| use mel spectrogram instead of EnCodec latent (ablation) | UniWav · LibriSpeech speech synthesis, 200k steps | ASR-WER: 2.5 -> 2.5 | No clear effect |
| use self-distillation without clustering (ablation) | UniWav encoder-decoder pre-training · pre-training stability | training stability | Hurt |
| distill from a pre-trained HuBERT (ablation) | UniWav encoder-decoder pre-training · pre-training stability | training stability | Hurt |
| reconstruct discrete EnCodec codes instead of using Flow Matching (ablation) | UniWav decoder · speech generation | generation performance | Hurt |
| add residual quantization to KM1024 semantic tokens (ablation) | UniWav speech tokenization · LibriSpeech test, tokenization | ASR-WER: 7.1 -> 5.6 | Helped |
Surprising findings
- Discriminative recognition benefits from the generative decoder only when the encoder is deep enough (24 layers); a shallow encoder is hurt by adding the decoder.
- Encoder depth matters more than decoder depth for generation quality, so the decoder size becomes less important as encoder depth grows.
Limitations
- Recognition performance trails state-of-the-art self-supervised models by about 0.5% WER, which the authors attribute to the cost of enabling generation.
- Evaluated only on English audiobook speech (LibriLight/LibriSpeech); generality to other languages and audio domains is untested.
- Speaker information is dropped at low bitrates, and speaker similarity lags behind codecs that retain speaker info.
Experimental setup
Datasets: LibriLight LibriSpeech
Benchmarks: LibriSpeech test-other LibriSpeech test-clean TTS SUPERB
Metrics: word error rate ASR-WER speaker similarity UTMOS ViSQOL phone error rate speaker identification accuracy MOS
Baselines: HuBERT WavLM data2vec 2.0 VALL-E Voicebox SpeechFlow SpeechTokenizer Unit-HiFiGAN
Builds on: DinoSR SpeechFlow Flow Matching EnCodec HuBERT
Closest papers by meaning
- Generative Pre-training for Speech with Flow Matching (ICLR 2024 · similarity 0.88)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.88)
- A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.87)
- Metis: A Foundation Speech Generation Model with Masked Generative Pre-training (NeurIPS 2025 · similarity 0.87)
- Scaling Speech-Text Pre-training with Synthetic Interleaved Data (ICLR 2025 · similarity 0.86)
- Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.