Research map / Video / Video generation
MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a pretrained one-shot NeRF-based person-agnostic 3D talking-face model to a target identity via tri-plane inversion plus LoRA (SD-Hybrid), and drives it with a flow-matching audio-to-motion model that mimics talking style in-context from a reference clip.
Problem
Personalized talking face generation usually requires training a separate NeRF per identity, which is slow, needs long training data, and generalizes poorly to out-of-domain audio and poses.
Approach
Use a pretrained person-agnostic 3D talking-face renderer (Real3D-Portrait) as a base, adapt it per identity with a learnable tri-plane (static) and LoRA layers (dynamic), and generate personalized motion with a flow-matching audio-to-motion model that takes a reference audio-motion pair as in-context style prompt, with a SyncNet lip-sync loss and classifier-free guidance on the style prompt.
Key results
- MimicTalk reaches CSIM 0.837, PSNR 31.72, FID 29.94, AED 0.098 and SyncNet 8.072, outperforming RAD-NeRF, GeneFace, ER-NeRF and StyleTalk on the reported metrics.
- Adaptation takes 2,000 iterations (about 0.26 h, 8.239 GB GPU memory on one A100), reported as 47x faster than RAD-NeRF.
- In a user study, MimicTalk gets the highest MOS on identity similarity (4.15), visual quality (4.22) and lip sync (4.13).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt the generic renderer with SD-Hybrid | MimicTalk versus strongest person-dependent baseline · 10 target-person videos | CSIM: 0.834 -> 0.837 | Helped |
| Adapt the generic renderer with SD-Hybrid | MimicTalk versus strongest person-dependent baseline · 10 target-person videos | PSNR: 31.45 -> 31.72 | Helped |
| Adapt the generic renderer with SD-Hybrid | MimicTalk versus strongest person-dependent baseline · 10 target-person videos | FID: 30.38 -> 29.94 | Helped |
| Adapt the generic renderer with SD-Hybrid | MimicTalk versus strongest baseline · 10 target-person videos | AED: 0.106 -> 0.098 | Helped |
| Adapt the generic renderer with SD-Hybrid | MimicTalk versus RAD-NeRF · 10 target-person videos | Training time (h): 13.22 -> 0.26 | Helped |
| Add tri-plane inversion (ablation) | SD-Hybrid adaptation · SD-Hybrid adaptation | CSIM: 0.823 -> 0.837 | Helped |
| Add LoRAs (ablation) | SD-Hybrid adaptation · SD-Hybrid adaptation | CSIM: 0.805 -> 0.837 | Helped |
| Use flow matching instead of a deterministic transformer (ablation) | ICS-A2M · ICS-A2M | L2 Landmark: 0.034 -> 0.026 | Helped |
| Use in-context style control instead of a style vector (ablation) | ICS-A2M · ICS-A2M | L2 Landmark: 0.031 -> 0.026 | Helped |
| Use in-context style control instead of a style encoder (ablation) | ICS-A2M · ICS-A2M | L2 Landmark: 0.029 -> 0.026 | Helped |
| Add sync loss (ablation) | ICS-A2M · ICS-A2M | Sync loss: 0.536 -> 0.423 | Helped |
Limitations
- Hair and torso are modeled rigidly and occasionally produce artifacts.
- Inference runs at about 15 FPS on one A100, which limits speed.
- Only eye movement and hand gestures are not modeled; the authors note these as missing conditions.
Experimental setup
Datasets: VoxCeleb2 talking face videos of 10 identities (3 minutes each, from prior works)
Benchmarks: 10 three-minute target person videos
Metrics: CSIM PSNR FID AED SyncNet confidence APD L2 landmark error MOS CMOS training time
Baselines: RAD-NeRF GeneFace ER-NeRF StyleTalk SadTalker
Builds on: Real3D-Portrait flow matching LoRA SyncNet classifier-free guidance
Closest papers by meaning
- Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation (AAAI 2024 · similarity 0.91)
- InsTaG: Learning Personalized 3D Talking Head from Few-Second Video (CVPR 2025 · similarity 0.89)
- Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation (CVPR 2025 · similarity 0.89)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.88)
- Faces that Speak: Jointly Synthesising Talking Face and Speech from Text (CVPR 2024 · similarity 0.88)
- FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.