Research map / Vision-language and multimodal / Cross-modal learning
MM-TTS: Multi-Modal Prompt Based Style Transfer for Expressive Text-to-Speech Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MM-TTS, a TTS framework that accepts reference speech, emotional face images, or text descriptions as style prompts via a CLIP-aligned prompt encoder, SAConv style modules, and a Rectified Flow refiner, and releases the MEAD-TTS dataset.
Problem
Existing TTS style transfer relies on fixed emotion labels, reference speech, or text descriptions alone, so it cannot flexibly accept arbitrary style prompt modalities in one system.
Approach
Align speech, face-image and text style prompts into a unified style space using CLIP-based encoders with adapters and MSE alignment, transfer the style via predicted convolution kernels (SAConv) rather than SALN, and refine over-smoothed mel-spectrograms with a one-to-few-step rectified flow refiner.
Key results
- On intra-domain MEAD-TTS, MM-TTS reaches MOS 4.36 and SMOS 4.61 with SECS 0.956 and MCD 3.17, the best among compared systems.
- For face-prompt style transfer on MEAD-TTS, MM-TTS raises emotion accuracy to 0.659 from 0.651 for MM-StyleSpeech, and out-of-domain gender accuracy rises to 0.833 from 0.686.
- Reflow refiner with one Euler step matches the RK45 sampler quality (MOS 4.32 vs 4.36 intra-domain), outperforming DDPM refiners in quality and speed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MM-TTS instead of MM-StyleSpeech | parallel reference speech style transfer · MEAD-TTS intra-domain | SMOS: 4.49 $\pm$ 0.06 -> 4.61 $\pm$ 0.07 | Helped |
| Use MM-TTS instead of MM-StyleSpeech | parallel reference speech style transfer · Out-of-domain | SECS: 0.713 -> 0.728 | Mixed |
| Use MM-TTS instead of MM-StyleSpeech | face-prompt style transfer · MEAD-TTS intra-domain | MOS: 3.98 $\pm$ 0.05 -> 4.11 $\pm$ 0.06 | Helped |
| Use MM-TTS instead of MM-StyleSpeech | text-prompt style transfer · MEAD-TTS intra-domain | MOS: 3.99 $\pm$ 0.04 -> 4.19 $\pm$ 0.07 | Helped |
| Add SAConv instead of SALN (ablation) | MM-TTS · Out-of-domain parallel reference speech | SMOS: 3.81 $\pm$ 0.07 -> 4.03 $\pm$ 0.08 | Helped |
| Add the Reflow Refiner training stage (ablation) | MM-TTS · MEAD-TTS intra-domain parallel reference speech | MOS: 4.17 $\pm$ 0.05 -> 4.36 $\pm$ 0.06 | Helped |
| Pretrain on LJSpeech (ablation) | MM-TTS · Out-of-domain parallel reference speech | MOS: 3.61 $\pm$ 0.07 -> 3.95 $\pm$ 0.08 | Helped |
Surprising findings
- Removing LJSpeech pretraining hurts out-of-domain naturalness much more than in-domain, with out-of-domain MOS dropping from 3.95 to 3.61.
- The face-prompt emotion accuracy is low in absolute terms (about 0.26 out-of-domain), suggesting facial prompts convey emotion poorly out of domain.
Limitations
- MEAD-TTS is small, which limits real-world applicability as shown by out-of-domain results.
- The hand-designed text description template struggles with abstract or complex styles.
- Evaluation of face and text prompts relies on a small set of emotion and gender categories.
Experimental setup
Datasets: MEAD-TTS LJSpeech LibriTTS Oulu-CASIA MEAD
Benchmarks: MEAD-TTS LibriTTS Oulu-CASIA LJSpeech
Metrics: MOS SMOS SECS MCD AXY preference emotion accuracy gender accuracy
Baselines: MS-FastSpeech2 GenerSpeech IST-TTS StyleSpeech MM-StyleSpeech DDPM refiner
Builds on: StyleSpeech FastSpeech2 CLIP Rectified Flow Meta-StyleSpeech
Closest papers by meaning
- Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis (ICLR 2024 · similarity 0.87)
- StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis (AAAI 2024 · similarity 0.86)
- StyleStudio: Text-Driven Style Transfer with Selective Control of Style Elements (CVPR 2025 · similarity 0.85)
- Adaptive Prompt Routing for Arbitrary Text Style Transfer with Pre-trained Language Models (AAAI 2024 · similarity 0.85)
- FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles (AAAI 2025 · similarity 0.85)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.