Research map / Point clouds / Point cloud classification
PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PointTalk, a 3D Gaussian Splatting talking-head method conditioned on an audio-driven dynamic lip point cloud, with a dynamic difference encoder and audio-point enhancement module.
Problem
Audio-driven talking head synthesis from a few minutes of training video suffers from poor audio-lip synchronization and visual quality due to limited training data in radiance-field methods.
Approach
Generate a dynamic lip point cloud from speech audio (Audio2Point), encode its topology with a multi-resolution hash grid and frame-to-frame feature differences, and fuse audio and point features with cross-modal contrastive learning, external attention, and AdaIN-style adaptive 3DGS deformation.
Key results
- PointTalk achieves PSNR 32.770, LPIPS 0.0337, LMD 2.818 in head reconstruction, the best among compared methods on these metrics.
- On Audio A and Audio B, PointTalk reaches LSE-C 6.427 and 7.018, second-best behind Wav2Lip among compared methods.
- Runs at about 85 FPS with roughly 1 hour training, close to TalkingGaussian's 90 FPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PointTalk instead of TalkingGaussian | talking-head synthesis · Head reconstruction | PSNR: 32.398 -> 32.770 | Helped |
| Use PointTalk instead of TalkingGaussian | talking-head synthesis · Head reconstruction | LPIPS: 0.0355 -> 0.0337 | Helped |
| Use PointTalk instead of ER-NeRF | talking-head synthesis · Head reconstruction | FID: 6.927 -> 7.331 | Hurt |
| Use PointTalk instead of TalkingGaussian | talking-head synthesis · Head reconstruction | LMD: 2.967 -> 2.818 | Helped |
| Use PointTalk instead of GeneFace++ | talking-head synthesis · Head reconstruction | LSE-D: 7.547 -> 7.383 | Helped |
| Use PointTalk instead of TalkingGaussian | talking-head synthesis · Head reconstruction | LSE-C: 6.516 -> 7.165 | Helped |
| Replace the lip point-cloud hash grid with a GNN (ablation) | PointTalk · Head reconstruction | PSNR: 32.704 -> 31.011 | Hurt |
| Replace the lip point-cloud hash grid with a tri-plane representation (ablation) | PointTalk · Head reconstruction | PSNR: 32.704 -> 32.390 | Hurt |
| Set hash-grid L=32 and F=1 instead of L=8 and F=4 (ablation) | PointTalk · Head reconstruction | PSNR: 32.704 -> 32.515 | Hurt |
| Set hash-grid L=16 and F=2 instead of L=8 and F=4 (ablation) | PointTalk · Head reconstruction | PSNR: 32.704 -> 32.662 | Hurt |
| Set hash-grid L=4 and F=8 instead of L=8 and F=4 (ablation) | PointTalk · Head reconstruction | PSNR: 32.704 -> 32.617 | Hurt |
| Add the dynamic difference encoder (DDE) (ablation) | PointTalk · Head reconstruction | LSE-D: 7.649 -> 7.367 | Helped |
Surprising findings
- Wav2Lip, a 2D one-shot method, achieves the best LSE-D and LSE-C on the self-driven evaluation, outperforming all 3D methods on lip-sync metrics.
Limitations
- Lip-sync metrics on out-of-distribution audio (LSE-D 8.3-8.4) remain weaker than 2D one-shot Wav2Lip.
- Relies on per-identity training video of a few minutes and a separately trained Audio2Point mesh-based module.
Experimental setup
Datasets: high-definition speaking video clips (English and French) sourced from prior work
Benchmarks: head reconstruction setting on self-driven test split lip synchronization with out-of-distribution Audio A and Audio B
Metrics: PSNR LPIPS FID LMD LSE-D LSE-C training time FPS
Baselines: Wav2Lip TalkLip DINet AD-NeRF RAD-NeRF GeneFace++ ER-NeRF TalkingGaussian
Builds on: 3D Gaussian Splatting TalkingGaussian Instant-NGP AdaIN external attention
Closest papers by meaning
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.91)
- Hierarchically Controlled Deformable 3D Gaussians for Talking Head Synthesis (AAAI 2025 · similarity 0.91)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.89)
- SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis (CVPR 2024 · similarity 0.89)
- MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation (AAAI 2025 · similarity 0.89)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.