Research map / 3D vision / Gaussian splatting
Monocular and Generalizable Gaussian Talking Head Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MGGTalk, a generalizable 3DGS talking head framework trained only on monocular data that uses depth-based point clouds with facial symmetry and a two-stage Gaussian prediction to generalize to unseen identities in real time.
Problem
One-shot talking head animation with 3D Gaussian Splatting typically needs multi-view data or person-specific training; monocular single-image input leaves geometry and appearance incomplete, especially in occluded regions.
Approach
Combines monocular depth estimation (Geowizard) with BINI surface reconstruction, a refinement UNet, and mirrored point clouds with voxel filtering to build complete 3D positions, then predicts visible-region Gaussians first and uses them to guide symmetric invisible-region Gaussians, enabling multi-view consistent generalizable talking heads trained on monocular data only.
Key results
- Video-driven self-reenactment on HDTF: PSNR 32.40 and FID 18.95, beating the best baselines (Real3DPortrait PSNR 31.62, Portrait4D-v2 FID 36.57).
- Audio-driven on HDTF: SSIM 0.731 and FID 18.73, best among all methods; LSE-C 7.68 and LSE-D 6.91 are second only to Wav2Lip.
- Runs at over 40 FPS on a single RTX 4090 after about 4 days of training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add geometry refinement network (ablation) | MGGTalk DSGR · HDTF self-reenactment | PSNR: 29.98 -> 32.40 | Helped |
| add symmetry operation (ablation) | MGGTalk DSGR · HDTF self-reenactment | PSNR: 31.24 -> 32.40 | Helped |
| add Gaussian/voxel filtering (ablation) | MGGTalk DSGR · HDTF self-reenactment | PSNR: 29.56 -> 32.40 | Helped |
| use point-cloud input to SGP instead of expression features (ablation) | MGGTalk SGP · HDTF self-reenactment | PSNR: 30.78 -> 32.40 | Helped |
| add two-stage symmetric Gaussian decoder (ablation) | MGGTalk SGP · HDTF self-reenactment | PSNR: 32.12 -> 32.40 | Helped |
| provide visible-region Gaussian parameters to the symmetric Gaussian decoder (ablation) | MGGTalk SGP · HDTF self-reenactment | PSNR: 31.68 -> 32.40 | Helped |
| add Gaussian densification (ablation) | MGGTalk · HDTF self-reenactment | PSNR: 30.32 -> 32.40 | Helped |
| apply geometric symmetry in DSGR before deformation rather than in SGP (ablation) | MGGTalk · talking-head video generation | visual stability across frames | Helped |
Surprising findings
- Applying symmetry inside the SGP module (after expression-driven deformation) causes cross-frame instability, so symmetry is placed in DSGR before deformation.
Limitations
- Invisible-region geometry relies on a symmetry assumption, so strongly asymmetric content is only partially handled.
- Lip-sync metrics (LSE-C/LSE-D) trail Wav2Lip, which uses SyncNet joint training.
Experimental setup
Datasets: HDTF NeRSemble
Benchmarks: HDTF NeRSemble-Mono
Metrics: PSNR SSIM FID AED APD LMD LSE-C LSE-D
Baselines: Styleheat DaGAN ROME OTAvatar Real3DPortrait Portrait4D-v2 Wav2Lip IP-LAP SadTalker AniTalker EchoMimic NeRFFaceSpeech
Builds on: 3D Gaussian Splatting Geowizard Bilateral Normal Integration Deep3DFaceReconstruction SadTalker
Closest papers by meaning
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.84)
- Hierarchically Controlled Deformable 3D Gaussians for Talking Head Synthesis (AAAI 2025 · similarity 0.84)
- PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis (AAAI 2025 · similarity 0.83)
- AnyTalk: Multi-modal Driven Multi-domain Talking Head Generation (AAAI 2025 · similarity 0.83)
- Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation (CVPR 2025 · similarity 0.82)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.