Research map / 3D vision / Face and shape
Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mimic disentangles speaking style and content latent spaces from 3D facial motion, enabling arbitrary-subject style encoding for speech-driven 3D facial animation, and introduces the 3D-HDTF dataset.
Problem
Speech-driven 3D facial animation mostly ignores subject-specific speaking style; existing one-hot identity styles cannot represent informative styles or generalize to unseen subjects without slow adaptation.
Approach
Learn separate style and content latent spaces from facial motion using a style encoder, content encoder, and audio encoder, with four constraints (auxiliary style classifier, gradient-reversal inverse classifier, content contrastive loss with audio, paired latent cycle losses), and inject style into the decoder via style-adaptive layer norm so any reference motion sequence can set the style.
Key results
- On 3D-HDTF Test-A, Mimic reduces FVE to 0.55 (x1e-6 mm) vs 1.02 for CodeTalker and reaches SCS 0.995 vs 0.965.
- On 3D-HDTF Test-B, Mimic outperforms Imitator (FVE 0.60 vs 0.85, SCS 0.992 vs 0.972).
- User study: participants preferred Mimic over Imitator for speaking style at 58.68% on 3D-HDTF Test-B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Mimic instead of CodeTalker | 3D-HDTF Test-A speech-driven facial animation · 3D-HDTF Test-A | FVE (×10^-6 mm): 1.02 -> 0.55 | Helped |
| Use Mimic instead of Imitator | unseen-subject speech-driven facial animation · 3D-HDTF Test-B | FVE (×10^-6 mm): 0.85 -> 0.60 | Helped |
| Add the content encoder during training (ablation) | Mimic · 3D-HDTF-Test-A | FVE (×10^-6 mm): 0.664 -> 0.551 | Helped |
| Add the auxiliary style classifier (ablation) | Mimic · 3D-HDTF-Test-A | SCS: 0.985 -> 0.995 | Helped |
| Add the auxiliary inverse classifier (ablation) | Mimic · 3D-HDTF-Test-A | FVE (×10^-6 mm): 0.620 -> 0.551 | Helped |
| Add the content contrastive loss (ablation) | Mimic · 3D-HDTF-Test-A | FVE (×10^-6 mm): 0.635 -> 0.551 | Helped |
| Add the style cycle loss (ablation) | Mimic · 3D-HDTF-Test-A | SCS: 0.987 -> 0.995 | Helped |
| Add the content cycle loss (ablation) | Mimic · 3D-HDTF-Test-A | FVE (×10^-6 mm): 0.633 -> 0.551 | Helped |
Surprising findings
- Removing the content encoder during training degrades all metrics, indicating that the content space with audio features alone carries too little motion information.
Limitations
- Relies on high-quality 3D face data from capture or accurate reconstruction.
- 3D-HDTF ground truth is pseudo meshes generated by SPECTRE rather than captured data.
Experimental setup
Datasets: 3D-HDTF HDTF VOCASET BIWI
Benchmarks: 3D-HDTF Test-A 3D-HDTF Test-B VOCA-Test BIWI-Test-B
Metrics: FVE LVE LDTW LDD SCS user study preference
Baselines: VOCA MeshTalk FaceFormer CodeTalker Imitator
Builds on: FaceFormer CodeTalker Imitator wav2vec 2.0 CLIP
Closest papers by meaning
- MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes (NeurIPS 2024 · similarity 0.91)
- Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks Methods and Applications (CVPR 2024 · similarity 0.89)
- ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.89)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.89)
- Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation (NeurIPS 2026 · similarity 0.88)
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.