Research map / Image generation / Editing and style transfer
Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ID-FaceVC, a zero-shot face-based voice conversion method using query-based contrastive learning on CLIP facial features and mutual-information-based decoupling of content and speaker features, accepting audio or text input with emotion and speed control.
Problem
Zero-shot face-based voice conversion requires well-aligned face-voice identity embeddings and clean decoupling of content from speaker identity in audio; prior methods use general facial features and hard low-pass filtering or implicit supervision.
Approach
Uses a Self-Adaptive Face-Prompted QFormer over frozen CLIP visual features, trained with contrastive and identity supervision to extract speaker-relevant facial features; decouples content and speaker style with a variational mutual-information upper bound; and adds text input via a VITS base TTS with emotion/speed style control.
Key results
- ID-FaceVC with audio input achieves UTMOS 3.286 vs 3.023 for the next best (FVMVC) and WER 12.11% vs 16.67% for FaceVC.
- With text input, ID-FaceVC reaches WER 5.02% and CER 2.69%, versus 14.31% and 8.70% for FaceTTS.
- Ablation shows IAQ-CL raises UTMOS from 2.945 to 3.227 over the baseline without it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add IAQ-CL instead of static facial feature extraction (ablation) | ID-FaceVC · LRS3 test, audio input | UTMOS: 2.945 -> 3.227 | Helped |
| Add MIDD to IAQ-CL (ablation) | ID-FaceVC · LRS3 test, audio input | WER: 18.04% -> 12.70% | Helped |
| Add MIDD to IAQ-CL (ablation) | ID-FaceVC · LRS3 test, audio input | SECS: 0.726 -> 0.709 | Hurt |
| Add face identity supervision L_id-f (ablation) | ID-FaceVC with IAQ-CL and MIDD · LRS3 test, audio input | CER: 8.63% -> 7.97% | Helped |
| Add speech identity supervision L_id-s (ablation) | ID-FaceVC with IAQ-CL and MIDD · LRS3 test, audio input | WER: 12.70% -> 12.01% | Helped |
| Add both face and speech identity supervision losses (ablation) | ID-FaceVC with IAQ-CL and MIDD · LRS3 test, audio input | UTMOS: 3.266 -> 3.286 | Helped |
| Use ID-FaceVC instead of FaceVC | Face-based voice conversion with audio input · LRS3 test, audio input | UTMOS: 2.155 -> 3.286 | Helped |
| Use ID-FaceVC instead of SP-FaceVC | Face-based voice conversion with audio input · LRS3 test, audio input | SECS: 0.723 -> 0.713 | Hurt |
| Use ID-FaceVC instead of FaceTTS | Face-based voice conversion with text input · LRS3 test, text input | UTMOS: 2.102 -> 3.454 | Helped |
Surprising findings
- SP-FaceVC scores higher SECS (0.723) than ID-FaceVC, which the authors attribute to averaged timbre convergence in SP-FaceVC's outputs.
Limitations
- Some SECS metric trade-off: content-focused design slightly reduces speaker similarity versus SP-FaceVC.
- Evaluation limited to 16 test speakers from LRS3 TED talks.
Experimental setup
Datasets: LRS3
Benchmarks: LRS3
Metrics: UTMOS WER CER SECS SEC SED nMOS sMOS preference test
Baselines: FaceVC SP-FaceVC FVMVC FaceTTS Face2Speech
Builds on: FVMVC SP-FaceVC QFormer OpenVoice CLAP
Closest papers by meaning
- FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles (AAAI 2025 · similarity 0.88)
- Emotional Face-to-Speech (ICML 2025 · similarity 0.87)
- StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching (AAAI 2025 · similarity 0.87)
- SelfVC: Voice Conversion With Iterative Refinement using Self Transformations (ICML 2024 · similarity 0.87)
- Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement (ICLR 2025 · similarity 0.86)
- Faces that Speak: Jointly Synthesising Talking Face and Speech from Text (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.