Research map / Label-efficient and robust learning / Self-supervised and contrastive

Metis: A Foundation Speech Generation Model with Masked Generative Pre-training

NeurIPS 2025 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Self-supervised speech and brain decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Metis pre-trains a masked generative transformer on 300K hours of unlabeled speech over SSL tokens, then fine-tunes (full or LoRA) with task-specific conditions to perform zero-shot TTS, voice conversion, target speaker extraction, speech enhancement, and lip-to-speech.

Problem

Speech generation tasks (TTS, voice conversion, enhancement, extraction, lip-to-speech) are handled by task-specific expert models or multi-task AR models that need large paired data per task; a unified model that leverages unlabeled speech and adapts efficiently is lacking.

Approach

Two-stage generation: a task-specific masked generative model maps conditions to SSL tokens, pre-trained unconditionally with masked token prediction (with optional prompt) on large unlabeled speech; task conditions are added during lightweight fine-tuning (concatenation for non-frame-level, interpolated MLP adapter for frame-level), and a masked generative acoustic decoder maps SSL tokens to acou

Key results

• Metis-TTS LoRA 32 fine-tuned on 1K hours (32M trainable params) reaches comparable WER/SIM to baselines trained on 10-100x more data; full fine-tuning on 10K hours gives WER 2.28 on SeedTTS test-en vs 2.47 for MaskGCT.
• Metis-VC LoRA 16 reaches SIM 0.55 on VCTK vs 0.38 for best baseline (Vevo/HierSpeech++).
• Metis-SE achieves NISQA 4.57 on DNS2020 with reverb and state-of-the-art SIG/BAK/OVRL across the three DNS test sets; Metis-Omni reduces TSE WER to 2.70 with text guidance.

Measured effects reported in the paper

Surprising findings

• Pre-trained model without fine-tuning generates speech that mimics prompt prosody and timbre but lacks intelligibility, producing random word concatenations.
• Metis-TSE with LoRA achieves higher NISQA than ground truth on LibriMix and EmiliaMix.

Limitations

• Relies on two separate discrete representations (SSL and acoustic tokens) rather than a unified audio representation.
• Fine-tuned models still face WER gaps versus some baselines (e.g., WeSep, Vevo on WER in TSE and VC).
• Few-shot task learning and in-context adaptation to new tasks are not explored.

Experimental setup

Datasets: Emilia LibriTTS VCTK LibriMix LRW LRS2 LRS3 DNS Challenge noise and RIR data WHAM! DEMAND SeedTTS test sets

Benchmarks: SeedTTS test-en SeedTTS test-zh LibriSpeech test-clean VCTK LibriMix test EmiliaMix test DNS2020 LRS2 LRS3

Metrics: WER SIM DNSMOS NISQA SIG BAK OVRL QMOS SMOS

Baselines: VALL-E VoiceCraft CosyVoice XTTS-v2 MaskGCT VoiceBox HierSpeech++ LM-VC UniAudio Vevo WeSep VoiceFilter

Builds on: MaskGCT SoundStorm DAC Vocos Emilia

Closest papers by meaning

• UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation (ICLR 2025 · similarity 0.87)
• Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.87)
• GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling (ICLR 2025 · similarity 0.86)
• Generative Pre-training for Speech with Flow Matching (ICLR 2024 · similarity 0.86)
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.86)
• Scaling Speech-Text Pre-training with Synthetic Interleaved Data (ICLR 2025 · similarity 0.86)