Research map / Image generation / Editing and style transfer
DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces decoupled denoising diffusion models (DDDMs) with one denoiser per speech attribute (content/filter and pitch/source) plus a prior mixup training scheme, applied to zero/one-shot voice conversion (DDDM-VC).
Problem
Single-denoiser diffusion models struggle to control individual attributes (linguistic content, intonation, timbre) for style transfer, and train-inference mismatch hurts voice conversion in non-parallel, zero/few-shot settings.
Approach
Subdivide the diffusion denoiser into attribute-specific denoisers, each with a disentangled source/filter Mel prior from a source-filter encoder, and during training use randomly converted representations as priors (prior mixup) so the decoder learns to convert rather than only reconstruct.
Key results
- Zero-shot VC on VCTK: DDDM-VC-Base (6 iterations) reaches EER 4.00 and SECS 0.843 vs DiffVC's 10.50 and 0.817 (many-to-many on LibriTTS, Table 2).
- DDDM-VC-Small with 21M parameters outperforms DiffVC with 123M parameters on speaker similarity and intelligibility metrics in many-to-many VC.
- One-shot fine-tuning of DDDM-VC-Base on VCTK improves SECS from 0.856 to 0.910 and EER from 6.25 to 0.81.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DDDM-VC-Base instead of DiffVC | voice conversion · LibriTTS many-to-many | EER: 10.50 -> 4.00 | Helped |
| Use DDDM-VC-Base instead of DiffVC | voice conversion · VCTK zero-shot | EER: 25.30 -> 6.25 | Helped |
| Fine-tune on one sample per speaker for 500 steps | DDDM-VC-Base · VCTK one-shot speaker adaptation | EER: 6.25 -> 0.81 | Helped |
| Add Prior Mixup (ablation) | DDDM-VC-Base · LibriTTS many-to-many | EER: 7.99 -> 4.24 | Helped |
| Add the disentangled denoiser (ablation) | DDDM-VC-Base · LibriTTS many-to-many | EER: 9.75 -> 4.24 | Helped |
| Add normalized F0 conditioning (ablation) | DDDM-VC-Base · LibriTTS many-to-many | EER: 10.25 -> 4.24 | Helped |
| Add the data-driven prior (ablation) | DDDM-VC-Base · LibriTTS many-to-many | EER: 19.25 -> 4.24 | Helped |
| Use Prior Mixup (ablation) | DDDM-VC decoder using reconstructed Mel as prior · VCTK zero-shot | EER: 7.10 -> 48.34 | Hurt |
Surprising findings
- Without prior mixup, the diffusion decoder mainly acts as speech enhancement when given a wrong prior, with EER rising to 48.34 in the wrong-prior test; with prior mixup it still converts voice style (EER 7.10).
- Scaling the model without prior mixup hurts voice conversion performance, as the large model learns to ignore conditioning.
Limitations
- Naturalness slightly decreases with prior mixup because target rhythm is not modeled on fixed-length input.
- Fine-tuning for one-shot adaptation overfits and increases CER.
- Relies on a pretrained XLS-R, F0 extraction, and pitch VQ-VAE pipeline.
Experimental setup
Datasets: LibriTTS VCTK VoxCeleb2
Benchmarks: LibriTTS VCTK
Metrics: nMOS sMOS CER WER EER SECS MCD Params Real-time factor
Baselines: AutoVC VoiceMixer SR (Polyak et al. 2021) DiffVC
Builds on: DiffVC score-based SDE diffusion ML-SDE solver eDiff-i XLS-R
Closest papers by meaning
- DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation (ICLR 2024 · similarity 0.88)
- Cauchy Diffusion: A Heavy-tailed Denoising Diffusion Probabilistic Model for Speech Synthesis (AAAI 2025 · similarity 0.88)
- Predictive–Generative Drift Decomposition for Speech Enhancement and Separation (NeurIPS 2026 · similarity 0.88)
- DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis (ICML 2025 · similarity 0.87)
- Shallow Diffusion for Fast Speech Enhancement (Student Abstract) (AAAI 2024 · similarity 0.87)
- Can Diffusion Models Disentangle? A Theoretical Perspective (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.