Research map / 3D vision / NeRF and radiance fields
InsTaG: Learning Personalized 3D Talking Head from Few-Second Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InsTaG, a 3DGS-based personalized 3D talking head framework that pre-trains an identity-free universal motion field on long videos and adapts it to a new identity from ~5 seconds of video.
Problem
Person-specific radiance-field talking heads need lots of data and long training per identity, while image-referred few-shot methods lose personalization and efficiency; the goal is fast, high-quality personalized 3D talking heads from few-second videos.
Approach
Pre-train a person-specific 3DGS synthesizer with per-video Personalized Fields and a shared Universal Motion Field (with a negative contrast loss separating personalized motion), then adapt to a new identity via a motion aligner, a face-mouth hook, and geometry regularization.
Key results
- With 5s training data, InsTaG reaches PSNR 28.86 (all image), LPIPS 0.039, LMD 3.167, and Sync-C 5.318, with 13 min adaptation and 82.5 FPS.
- At 20s training data, InsTaG achieves PSNR 30.352 and AUE-L 0.710, outperforming ER-NeRF, TalkingGaussian, DFRF and GeneFace.
- In a user study with 10 raters, InsTaG was rated best on image quality, identity preserving and video realness, and second-best on lip-sync behind MimicTalk.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InsTaG instead of TalkingGaussian | 5s self-reconstruction · Self-reconstruction, 5s | PSNR: 28.321 -> 28.863 | Helped |
| Use InsTaG instead of TalkingGaussian | 5s self-reconstruction · Self-reconstruction, 5s | Sync: 3.556 -> 5.318 | Helped |
| Use InsTaG instead of MimicTalk | 5s self-reconstruction · Self-reconstruction, 5s | Sync-C: 6.926 -> 5.318 | Hurt |
| Use InsTaG instead of MimicTalk | 5s self-reconstruction · Self-reconstruction, 5s | Training time: 16 min -> 13 min | Helped |
| Add Personalized Fields during pre-training (ablation) | InsTaG pre-training · 5s self-reconstruction ablation | Sync-C: 2.019 -> 4.816 | Helped |
| Add Negative Contrast Loss (ablation) | InsTaG pre-training with Personalized Fields · 5s self-reconstruction ablation | Sync-C: 4.816 -> 5.124 | Helped |
| Add Motion Aligner (ablation) | InsTaG adaptation · 5s self-reconstruction ablation | PSNR: 28.59 -> 28.69 | Helped |
| Add Face-Mouth Hook (ablation) | InsTaG adaptation with Personalized Fields and Negative Contrast Loss · 5s self-reconstruction ablation | PSNR: 28.59 -> 28.66 | No clear effect |
| Add both Motion Aligner and Face-Mouth Hook (ablation) | InsTaG adaptation · 5s self-reconstruction ablation | LMD: 3.223 -> 3.167 | Helped |
| Replace a single branch with face-mouth decomposition (ablation) | InsTaG mouth-motion representation · 5s mouth-motion ablation | PSNR: 28.59 -> 28.69 | Helped |
| Add Face-Mouth Hook to face-mouth decomposition (ablation) | InsTaG mouth-motion representation · 5s mouth-motion ablation | PSNR: 28.69 -> 28.86 | Helped |
Surprising findings
- Image-referred one-shot methods (EAT, Real3DPortrait) reach high Sync scores but are far slower, and MimicTalk, a LoRA-adapted large backbone, is slow at inference (8.2 FPS).
- Pre-training with multiple identities in a shared person-specific model without identity separation hurts synchronization, motivating the personalized fields.
Limitations
- Some baselines fail on limited data, so the comparison uses 20s of training data per method for cross-domain tests.
- The pre-training corpus is small (5 long videos), and the reliance on a monocular geometry estimator for regularization is a dependency.
Experimental setup
Datasets: long-video corpus of 5 speaking videos from GeneFace/ER-NeRF sets (pre-training) publicly released sets from DFRF and GeneFace (4 test identities) NVP audio Real3DPortrait-style German female audio
Benchmarks: self-reconstruction setting (5s, 10s, 20s training) cross-domain setting (20s training)
Metrics: PSNR LPIPS SSIM LMD AUE-L AUE-U Sync-C Sync-E training time FPS
Baselines: EAT Real3DPortrait RAD-NeRF ER-NeRF GaussianTalker TalkingGaussian DFRF GeneFace MimicTalk
Builds on: TalkingGaussian 3D Gaussian Splatting ER-NeRF Face-mouth decomposition
Closest papers by meaning
- MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes (NeurIPS 2024 · similarity 0.89)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.88)
- Diffusion-based Realistic Listening Head Generation via Hybrid Motion Modeling (CVPR 2025 · similarity 0.87)
- AnyTalk: Multi-modal Driven Multi-domain Talking Head Generation (AAAI 2025 · similarity 0.87)
- VASA-3D: Lifelike Audio-Driven Gaussian Head Avatars from a Single Image (NeurIPS 2025 · similarity 0.87)
- IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular VideosC (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.