Research map / 3D vision / Face and shape
SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SyncTalk, a NeRF-based talking head method that synchronizes lip movements, facial expressions, head pose, and identity using a pre-trained audio-visual sync encoder, 3D blendshape expression control, bundle-adjusted head tracking, and a portrait refinement stage.
Problem
Speech-driven talking head synthesis lacks synchronization across lip movements, facial expressions, head poses, and identity; GAN methods lose identity consistency and NeRF methods produce mismatched lips, weak expressions, and unstable heads.
Approach
Combine a lip-sync-discriminator-supervised audio-visual encoder, 52-coefficient 3D blendshape expression features with masked attention for disentanglement, a SLAM-inspired keypoint tracking plus bundle adjustment head-pose stabilizer, and a Gaussian-blur portrait compositing step on top of tri-plane hash NeRF (ER-NeRF).
Key results
- SyncTalk (Portrait) reaches PSNR 37.40, LPIPS 0.0113 and FID 2.71 on head reconstruction, beating ER-NeRF (PSNR 32.52, LPIPS 0.0334, FID 5.29).
- On out-of-distribution audio, SyncTalk achieves LSE-D 7.72 / LSE-C 6.67 on Audio A, versus 8.50 / 5.70 for the best baseline DINet.
- Rendering runs at 50 FPS at 512x512 on an RTX 3090 with the Portrait-Sync Generator, versus 25 FPS source video rate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the Audio-Visual Encoder with HuBERT (ablation) | SyncTalk · May | LMD: 2.8032 -> 3.3961 | Hurt |
| Replace Facial Animation Capture with ER-NeRF’s blink module (ablation) | SyncTalk · May | PSNR: 37.311 -> 30.273 | Hurt |
| Remove Facial-Aware Masked-Attention (ablation) | SyncTalk · May | LPIPS: 0.0121 -> 0.0165 | Hurt |
| Remove Head-Sync Stabilizer (ablation) | SyncTalk · May | PSNR: 37.311 -> 28.984 | Hurt |
| Remove Portrait-Sync Generator (ablation) | SyncTalk · Head reconstruction | PSNR: 37.4017 -> 35.3542 | Hurt |
| Use SyncTalk instead of ER-NeRF | Head reconstruction · Head reconstruction | LPIPS: 0.0334 -> 0.0113 | Helped |
| Use SyncTalk instead of the strongest shown baseline | Audio A · Audio A | LSE-C: 5.7449 -> 6.6659 | Helped |
Surprising findings
- Removing the Head-Sync Stabilizer caused the largest ablation drop (PSNR 37.31 to 28.98), exceeding the effect of removing the audio-visual encoder.
- SyncTalk without the Portrait-Sync Generator scored lower LMD-based sync-related accuracy than the full model only by small margins, and its LSE-C (8.13) exceeds the full portrait mode (8.03), suggesting the refinement step does not improve lip sync confidence.
Limitations
- Evaluated on a small set of per-subject videos from prior work; sync gains measured with a few metrics and a 35-person user study.
- Per-subject training is required (about 2 hours on an RTX 3090 per video).
Experimental setup
Datasets: LRS2 AD-NeRF/ER-NeRF/GeneFace talking head video sequences
Benchmarks: AD-NeRF test videos (English and French)
Metrics: PSNR LPIPS MS-SSIM FID NIQE BRISQUE LMD AUE LSE-C LSE-D
Baselines: Wav2Lip VideoReTalking DINet TalkLip IP-LAP AD-NeRF RAD-NeRF GeneFace ER-NeRF
Builds on: ER-NeRF AD-NeRF Instant-NGP Wav2Lip lip sync discriminator
Closest papers by meaning
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.90)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.90)
- PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis (AAAI 2025 · similarity 0.89)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.89)
- MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation (AAAI 2025 · similarity 0.89)
- AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.