Research map / Image generation / Autoregressive and latent generation
DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a 450M latent diffusion TTS teacher into a 4-step student via improved distribution matching distillation (DMD2), then directly optimizes CTC (WER) and speaker-verification losses end-to-end, outperforming its teacher while running ~13.7x faster.
Problem
Diffusion-based zero-shot TTS is slow due to iterative sampling, prior distillation degrades quality, and non-differentiable components (duration predictors) and multi-step sampling block end-to-end optimization of perceptual metrics like WER and speaker similarity.
Approach
Distill the teacher with DMD2 into a few-step generator whose full sampling chain is differentiable, so CTC (WER) and speaker-verification losses on latent speech can be backpropagated through all components; the distilled student also shows beneficial mode shrinkage that improves conditional quality.
Key results
- DMOSpeech (N=4) reaches WER 1.94 and SIM 0.69 versus teacher WER 9.51 and SIM 0.55 on LibriSpeech test-clean, with RTF 0.07 vs 0.96 for the teacher (about 13.7x faster).
- Human listening tests show DMOSpeech (N=4) MOS-N 4.42 and SMOS-V 4.49, above the teacher (4.32 and 4.17) and NaturalSpeech 3 (4.24 MOS-N).
- Ablation: adding CTC loss cuts WER from 5.67 to 1.79; adding SV loss raises SIM to 0.70 in its SV-only variant; combining both gives the best balance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill the teacher to four sampling steps and optimize with CTC and SV losses | Teacher diffusion TTS · LibriSpeech test-clean (3,711 samples) | WER: 9.51 -> 1.94 | Helped |
| distill the teacher to four sampling steps and optimize with CTC and SV losses | Teacher diffusion TTS · LibriSpeech test-clean (3,711 samples) | SIM: 0.55 -> 0.69 | Helped |
| distill the teacher to four sampling steps | Teacher diffusion TTS · LibriSpeech test-clean (3,711 samples) | RTF: 0.96 -> 0.07 | Helped |
| add CTC loss (ablation) | DMD 2 with four sampling steps · LibriSpeech test-clean (3,711 samples) | WER: 5.67 -> 1.79 | Helped |
| add CTC loss (ablation) | DMD 2 with four sampling steps · LibriSpeech test-clean (80 samples) | SMOS-V: 3.69 -> 3.73 | No clear effect |
| add SV loss (ablation) | DMD 2 with four sampling steps · LibriSpeech test-clean (80 samples) | SMOS-V: 3.69 -> 4.35 | Helped |
| add SV loss (ablation) | DMD 2 with four sampling steps · LibriSpeech test-clean (3,711 samples) | WER: 5.67 -> 6.62 | Hurt |
| increase sampling from one step to four steps (ablation) | DMD 2 student · LibriSpeech test-clean (80 samples) | MOS-N: 3.11 ($\pm$ 0.05) -> 4.19 ($\pm$ 0.03) | Helped |
| reduce batch size from 96 to 16 (ablation) | DMOSpeech (N=4) · LibriSpeech test-clean (3,711 samples) | WER: 1.94 -> 3.38 | Hurt |
| reduce batch size from 96 to 16 (ablation) | DMOSpeech (N=4) · LibriSpeech test-clean (80 samples) | MOS-Q: 4.45 ($\pm$ 0.03) -> 4.30 ($\pm$ 0.03) | Hurt |
| use CTC and SV losses together instead of either loss alone (ablation) | DMD 2 with four sampling steps · LibriSpeech test-clean (80 samples) | MOS-N: 4.07 ($\pm$ 0.04) -> 4.27 ($\pm$ 0.03) | Mixed |
Surprising findings
- The distilled student outperforms its 128-step teacher in naturalness, similarity, and WER, attributed to mode shrinkage that removes teacher cutoff artifacts.
- Reduced conditional diversity (lower pitch CV under same text and prompt) is argued to be acceptable or even preferred for zero-shot TTS, since unconditional coverage is preserved.
Limitations
- Reduced speech diversity under identical text and prompt (mode shrinkage) is a trade-off.
- Trained only on English LibriLight data; scaling to more data and languages is untested.
- Speaker-similarity gains raise voice-cloning misuse and deepfake risks, and the SV model can be fooled by synthetic speech.
Experimental setup
Datasets: LibriLight CommonVoice LibriSpeech test-clean
Benchmarks: LibriSpeech test-clean
Metrics: MOS-N MOS-Q SMOS-V SMOS-S WER SIM RTF CV_f0
Baselines: NaturalSpeech 3 StyleTTS-ZS DiTTo-TTS VoiceCraft CLaM-TTS XTTS
Builds on: distribution matching distillation DMD2 DiTTo-TTS SimpleTTS DAC
Closest papers by meaning
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.88)
- DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors (ICLR 2025 · similarity 0.88)
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (ICML 2024 · similarity 0.88)
- Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations (AAAI 2024 · similarity 0.87)
- DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion (AAAI 2024 · similarity 0.87)
- TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.