Research map / Image generation / Autoregressive and latent generation
MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglement
Xinyue Yu, Youqing Fang, Pingyu Wu, Guoyang Ye, Wenbo Zhou, Weiming Zhang, Song Xiao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamic fusion (DyGate) (ablation) | MF-SpeechGenerator · ESD multi-factor compositional generation | SECS: 0.5551 -> 0.5685 | Improved |
| add dynamic fusion (DyGate) (ablation) | MF-SpeechGenerator · ESD multi-factor compositional generation | WER: 5.17% -> 4.76% | Improved |
| add dynamic fusion (DyGate) (ablation) | MF-SpeechGenerator · ESD multi-factor compositional generation | UTMOS, Log RMSE, Corr | No clear effect |
| add HSAN (ablation) | MF-SpeechGenerator · ESD multi-factor compositional generation | SECS: 0.1576 -> 0.5685 | Improved |
| add HSAN (ablation) | MF-SpeechGenerator · ESD multi-factor compositional generation | Corr: 0.64 -> 0.68 | Improved |
| add HSAN (ablation) | MF-SpeechGenerator · ESD multi-factor compositional generation | Log RMSE: 0.38 -> 0.34 | Improved |
| add HSAN (ablation) | MF-SpeechGenerator · ESD multi-factor compositional generation | WER | Hurt |
| add mutual information constraints (ablation) | MF-SpeechEncoder · ESD t-SNE of timbre and emotion | cluster separability | Improved |
| add contrastive learning (ablation) | MF-SpeechEncoder · ESD t-SNE of timbre and emotion | cluster separability | Improved |
| add prosody priors (ablation) | MF-SpeechEncoder · ESD t-SNE of timbre and emotion | cluster separability | Improved |
| use MF-SpeechEncoder instead of FACodec encoder | disentanglement · ESD factor disentanglement | Acc_e: 0.2343 -> 0.9296 | Improved |
| use MF-Speech instead of DDDM-VC | speech generation · ESD multi-factor compositional generation | WER: 11.67% -> 4.67% | Improved |
Closest papers by meaning
- Metis: A Foundation Speech Generation Model with Masked Generative Pre-training (NeurIPS 2025 · similarity 0.85)
- LLM-driven Multimodal and Multi-Identity Listening Head Generation (CVPR 2025 · similarity 0.84)
- CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching (NeurIPS 2025 · similarity 0.84)
- A Variational Framework for Improving Naturalness in Generative Spoken Language Models (ICML 2025 · similarity 0.84)
- GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling (ICLR 2025 · similarity 0.84)
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
