Research map / 3D vision / Face and shape
Towards High-fidelity 3D Talking Avatar with Personalized Dynamic Texture
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TexTalk4D, a 100-subject 4D dataset with 8K dynamic textures, and TexTalker, a quantized-latent diffusion model that jointly generates speech-driven facial motion and wrinkle textures with pivot-based style injection.
Problem
Speech-driven 3D talking head methods generate geometry but ignore dynamic facial texture (wrinkles, pore changes), and no large high-resolution dataset with aligned dynamic textures existed.
Approach
Represent vertex motion and texture variation as UV motion maps and wrinkle maps, compress each into a separate VQ latent space, and jointly diffuse motion-wrinkle latent offsets from per-subject style pivots (mean latent codes) so that style can be injected and controlled disentangledly.
Key results
- TexTalker reduces LVE to 1.49 and MVE to 2.34 (10^-2 mm) on TexTalk4D-Test-A, beating FaceFormer, CodeTalker, and FaceDiffuser on motion metrics.
- Dynamic texture generation reaches 44.13 PSNR / 0.985 SSIM / 0.0101 LPIPS versus 39.79 PSNR for static textures and 42.34 PSNR for Li et al.
- User study realism score 4.13 and consistency score 3.97 on TexTalk4D-Test-B, above Li et al. (3.91, 3.78) and static texture (3.10, 2.65).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TexTalker instead of FaceDiffuser | speech-driven facial motion generation · TexTalker4D-Test-A | LVE: 1.53 -> 1.49 | Helped |
| Use TexTalker instead of FaceDiffuser | speech-driven facial motion generation · TexTalker4D-Test-A | MVE: 2.38 -> 2.34 | Helped |
| Use TexTalker instead of CodeTalker | speech-driven facial motion generation · TexTalker4D-Test-A | FDD: 1.38 -> 1.20 | Helped |
| Use joint motion-texture generation instead of motion-only generation (ablation) | TexTalker · TexTalker4D-Test-A | LVE: 1.73 -> 1.49 | Helped |
| Use joint motion-texture generation instead of motion-only generation (ablation) | TexTalker · TexTalker4D-Test-A | FDD: 1.06 -> 1.20 | Hurt |
| Use joint motion-texture generation instead of wrinkle-only generation (ablation) | TexTalker · TexTalker4D-Test-A | PSNR: 43.87 -> 44.13 | Helped |
| Use separate motion and wrinkle codebooks instead of one joint codebook (ablation) | TexTalker · TexTalker4D-Test-A | PSNR: 43.45 -> 44.13 | Helped |
| Use separate motion and wrinkle codebooks instead of one joint codebook (ablation) | TexTalker · TexTalker4D-Test-A | FDD: 1.21 -> 1.20 | Hurt |
| Use pivot-based style injection instead of one-hot style conditioning (ablation) | TexTalker · TexTalker4D-Test-A | LVE: 1.70 -> 1.49 | Helped |
| Use pivot-based style injection instead of one-hot style conditioning (ablation) | TexTalker · TexTalker4D-Test-A | PSNR: 43.61 -> 44.13 | Helped |
| Use pivot-based style injection instead of one-hot style conditioning (ablation) | TexTalker · TexTalker4D-Test-A | FDD: 1.18 -> 1.20 | No clear effect |
Surprising findings
- Separate motion-only or texture-only generation gives better FDD (upper-face dynamics) than joint learning, while joint learning improves lip and overall accuracy.
- A single joint latent space for motion and wrinkles performs worse than independent codebooks on most metrics.
Limitations
- TexTalk4D lacks diversity in language, age, and race (Mandarin, Asian youths).
- Textures are learned at 512 resolution and upsampled to 8K via a super-resolution network, losing some detail.
- TexTalkr comparison with texture baselines is limited to one prior method (Li et al.) and a static baseline.
Experimental setup
Datasets: TexTalk4D Multiface
Benchmarks: TexTalk4D-Test-A TexTalk4D-Test-B
Metrics: LVE MVE FDD PSNR SSIM LPIPS realism score consistency score
Baselines: FaceFormer CodeTalker FaceDiffuser DiffPoseTalk Li et al. (texture image translation) Pix2PixHD
Builds on: VQGAN latent diffusion model DiffPoseTalk FaceDiffuser Topo4D
Closest papers by meaning
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.88)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.88)
- Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control (ICLR 2025 · similarity 0.87)
- Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation (AAAI 2024 · similarity 0.87)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.87)
- Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks Methods and Applications (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.