Research map / 3D vision / Gaussian splatting
TaoAvatar: Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents TaoAvatar, a lightweight 3DGS-based full-body talking avatar that bakes a StyleUnet teacher's non-rigid deformation into an MLP student plus learnable blend shapes, running at 90 FPS stereo on Apple Vision Pro.
Problem
Full-body talking avatars with fine-grained facial and body control and high detail are too heavy to render in real time on mobile and AR devices.
Approach
Bind 3D Gaussians to a personalized clothed SMPLX++ mesh as triangle textures, train a large StyleUnet teacher for pose-dependent deformation maps, distill these into a compact MLP student that deforms mesh vertices, and add learnable position/color blend shapes driven by head expression and body pose coefficients.
Key results
- Student model reaches 33.81 PSNR novel view on full-body talking vs 32.50 for AnimatableGS, while running at 156 FPS on RTX4090 at 1500x2000.
- Method renders stereo 2K at about 90 FPS on Apple Vision Pro, per the abstract and introduction.
- Student outperforms its teacher on the full-body talking novel-view metrics (33.81 vs 33.45 PSNR) while being far faster (156 vs 16 FPS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the clothed SMPLX++ template instead of SMPLX directly (ablation) | TaoAvatar · Full-body talking | PSNR: 28.47 -> 33.29 | Helped |
| Add mesh non-rigid deformation (ablation) | TaoAvatar · Full-body talking | PSNR: 32.10 -> 33.29 | Helped |
| Add Gaussian non-rigid compensation blend shapes (ablation) | TaoAvatar · Full-body talking | PSNR: 31.16 -> 33.29 | Helped |
| Use teacher supervision to train the student instead of direct MLP training (ablation) | TaoAvatar · Full-body talking | PSNR: 32.67 -> 33.29 | Helped |
Surprising findings
- The distilled student model outperforms the StyleUnet teacher on several metrics despite being much smaller.
Limitations
- Struggles to model flexible clothing deformation under exaggerated body poses that are out-of-distribution for training.
- Teacher StyleUnet is too resource-intensive for real-time mobile rendering.
Experimental setup
Datasets: TalkBody4D ActorHQ
Benchmarks: TalkBody4D ActorHQ
Metrics: PSNR SSIM LPIPS P2S Chamfer FPS
Baselines: GaussianAvatar 3DGS-Avatar MeshAvatar AnimatableGS
Builds on: 3D Gaussian Splatting SMPLX AnimatableGS NeuS2 SMPL pose corrective blend shapes
Closest papers by meaning
- SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting (CVPR 2024 · similarity 0.90)
- 3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting (CVPR 2024 · similarity 0.88)
- Human Gaussian Splatting: Real-time Rendering of Animatable Avatars (CVPR 2024 · similarity 0.87)
- MPMAvatar: Learning 3D Gaussian Avatars with Accurate and Robust Physics-Based Dynamics (NeurIPS 2025 · similarity 0.87)
- GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians (CVPR 2024 · similarity 0.87)
- CtrlAvatar: Controllable Avatars Generation via Disentangled Invertible Networks (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.