Research map / 3D vision / Face and shape
Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines three criteria (temporal synchronization, lip readability, expressiveness) for perceptually accurate 3D talking heads, builds a two-stage speech-mesh contrastive representation, uses it as a plug-in perceptual loss, and proposes MTM, PLRS, and SLCC metrics.
Problem
Speech-driven 3D talking head models optimize MSE/LVE, which does not capture perceptual alignment between speech characteristics (especially intensity) and lip movements, and existing datasets have limited intensity range.
Approach
Learn a speech-mesh representation by first training an audio-visual speech encoder on 2D LRS3 videos (MAE plus InfoNCE) and then aligning a 3D mesh encoder to this frozen speech space; use the resulting contrastive loss as a plug-and-play perceptual loss and as an evaluation metric alongside two physically grounded metrics (MTM, SLCC).
Key results
- Adding the speech-mesh perceptual loss to FaceFormer improves PLRS from 0.368 to 0.463 and LVE from 3.357 to 3.091 on VOCASET.
- On MEAD-3D, FaceFormer trained with MEAD-3D and our representation loss raises average SLCC from 0.07 (VOCASET-only) to 0.33 while keeping LVE near 1.1 (MEAD-3D data).
- Human study: participants preferred 100ms-asynchronous but expressive lip movements over precisely timed but low-expressiveness ones (82.6% vs 17.4%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the proposed perceptual loss to FaceFormer | FaceFormer · VOCASET test | PLRS: 0.368 -> 0.463 | Helped |
| Add the proposed perceptual loss to CodeTalker | CodeTalker · VOCASET test | PLRS: 0.381 -> 0.388 | Helped |
| Add the proposed perceptual loss to SelfTalk | SelfTalk · VOCASET test | PLRS: 0.414 -> 0.418 | Helped |
| Add the proposed perceptual loss to FaceFormer | FaceFormer · VOCASET test | LVE: 3.357 -> 3.091 | Helped |
| Add the proposed perceptual loss to FaceFormer | FaceFormer · VOCASET test | MTM: 53.6 -> 52.2 | Helped |
| Replace the proposed representation with 3D SyncNet (ablation) | FaceFormer · VOCASET test | PLRS: 0.463 -> 0.435 | Hurt |
| Remove the pre-trained 2D speech representation prior (ablation) | FaceFormer · VOCASET test | PLRS: 0.463 -> 0.400 | Hurt |
| Train SelfTalk with MEAD-3D in addition to VOCASET | SelfTalk · MEAD-3D test | SLCC Avg: 0.15 -> 0.31 | Helped |
| Train SelfTalk with MEAD-3D in addition to VOCASET | SelfTalk · MEAD-3D test | PLRS: 0.294 -> 0.216 | Hurt |
| Add the proposed perceptual loss after training with MEAD-3D | SelfTalk + MEAD-3D · MEAD-3D test | PLRS: 0.216 -> 0.230 | Helped |
Surprising findings
- Participants preferred expressive lip movements with 100ms asynchrony over precisely synchronized but less expressive ones, suggesting expressiveness matters more than strict timing.
- Adding the MEAD-3D data alone degraded lip synchronization metrics, requiring the perceptual loss to counterbalance.
Limitations
- Temporal synchronization is assessed via a proxy (DDTW) rather than ground-truth onset annotations.
- Pseudo ground-truth 3D meshes come from monocular reconstruction (SPECTRE, SMIRK) rather than real 3D scans.
- Adding MEAD-3D alone degrades lip synchronization metrics in several base models.
Experimental setup
Datasets: LRS3 LRS3-3D MEAD-3D VOCASET BIWI MEAD
Benchmarks: VOCASET MEAD-3D
Metrics: MTM LVE PLRS SLCC level-wise SLCC
Baselines: FaceFormer CodeTalker SelfTalk 3D SyncNet MultiTalk
Builds on: InfoNCE masked autoencoder AV-HuBERT SyncNet CLIP
Closest papers by meaning
- PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis (AAAI 2025 · similarity 0.91)
- SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis (CVPR 2024 · similarity 0.90)
- Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation (NeurIPS 2026 · similarity 0.90)
- ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.89)
- Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks Methods and Applications (CVPR 2024 · similarity 0.89)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.