Research map / Image generation / Autoregressive and latent generation
Scaling Speech Tokenizers with Diffusion Autoencoders
Yuancheng Wang, Zhenyu Tang, Yun Wang, Arthur Hinsvark, Yingru Liu, Yinghao Aaron Li, Kainan Peng, Junyi Ao, Mingbo Ma, Mike Seltzer, Qing He, Xubo Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CTC semantic regularization (weight 0.1) (ablation) | SiTok (12.5 Hz, 1 codebook) · SeedTTS test-en | WER (%): 33.0 -> 4.06 | Improved |
| add CTC semantic regularization (ablation) | SiTok · LLM ASR | WER: 29.4 -> 4.95 | Improved |
| raise CTC weight from 0.1 to 1 (ablation) | SiTok · SeedTTS test-en | WER (%): 4.06 -> 10.1 | Hurt |
| use diffusion loss instead of L1 regression loss (ablation) | SiTok · SeedTTS test-en | WER (%): 4.66 -> 4.06 | Improved |
| train diffusion end-to-end rather than finetuning only the decoder on a regression model (ablation) | SiTok · SeedTTS test-en | WER (%): 5.73 -> 4.06 | Improved |
| add token classifier-free guidance (ablation) | SiTok (CN=1) · SeedTTS test-en | WER (%): 4.06 -> 3.34 | Improved |
| add decoder finetuning (ablation) | SiTok (CN=1) · SeedTTS test-en | speaker similarity: 0.641 -> 0.682 | Improved |
| increase codebook size from 2^13 to 2^17 (ablation) | SiTok · SeedTTS test-en | WER (%): 5.48 -> 3.94 | Improved |
| increase number of RVQ codebooks from 1 to 8 (ablation) | SiTok · SeedTTS test-en | WER (%): 4.30 -> 2.50 | Improved |
| increase frame rate from 6.25 Hz to 12.5 Hz (ablation) | SiTok · SeedTTS test-en | WER (%): 23.0 -> 4.06 | Improved |
| scale model from L (1.12B) to XL (1.61B) (ablation) | SiTok · Understanding tasks | SV EER: 13.8 -> 14.7 | Hurt |
| scale model from S (0.63B) to XL (1.61B) (ablation) | SiTok · SeedTTS test-en | WER (%): 4.18 -> 3.84 | Improved |
Closest papers by meaning
- TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.89)
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.88)
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.88)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.87)
- DELTA-TTS: Adapting Autoregressive Model into a Diffusion Language Model for Text-to-Speech (NeurIPS 2026 · similarity 0.87)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
