Research map / 3D vision / Face and shape
Vid2Avatar-Pro: Authentic Avatar from Videos in the Wild via Universal Prior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a universal prior of clothed humans from ~1000 multi-view performance captures using front/back canonical maps with a U-Net predicting 3D Gaussians, then fine-tunes it on a monocular in-the-wild video to create animatable photorealistic avatars.
Problem
Creating photorealistic, animatable 3D human avatars from monocular in-the-wild videos is hard because limited pose and viewpoint coverage causes poor generalization to novel poses and overfitting to input views.
Approach
Replace SMPL or Laplacian priors with a learned universal prior: a cross-identity U-Net conditioned on spatially normalized canonical front and back texture/position maps predicts pose-dependent 3D Gaussian attributes, which is then personalized to a single video via inverse rendering with diffusion-based texture inpainting.
Key results
- On NeuMan interpolation synthesis, achieves 32.71 PSNR / 0.983 SSIM / 1.19 LPIPS, beating the best baseline ExAvatar (31.39 / 0.981 / 1.64).
- On MonoPerfCap extrapolation synthesis, achieves 31.97 PSNR / 0.981 SSIM / 1.37 LPIPS, versus 30.29 PSNR for ExAvatar.
- Ablation on number of training identities shows rendering quality rises from 31.28 PSNR (4 IDs) to 31.97 PSNR (1000 IDs).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Vid2Avatar-Pro instead of ExAvatar | monocular avatar reconstruction · NeuMan | PSNR: 31.39 -> 32.71 | Helped |
| use Vid2Avatar-Pro instead of GaussianAvatar | monocular avatar reconstruction · NeuMan | SSIM: 0.980 -> 0.983 | Helped |
| use Vid2Avatar-Pro instead of GaussianAvatar | monocular avatar reconstruction · NeuMan | LPIPS (×100): 1.24 -> 1.19 | Helped |
| use Vid2Avatar-Pro instead of ExAvatar | monocular avatar reconstruction · MonoPerfCap | PSNR: 30.29 -> 31.97 | Helped |
| use Vid2Avatar-Pro instead of ExAvatar | monocular avatar reconstruction · MonoPerfCap | SSIM: 0.979 -> 0.981 | Helped |
| use Vid2Avatar-Pro instead of ExAvatar | monocular avatar reconstruction · MonoPerfCap | LPIPS (×100): 2.19 -> 1.37 | Helped |
| add fine-tuning on monocular observations (ablation) | Vid2Avatar-Pro · MonoPerfCap | PSNR: 29.24 -> 31.97 | Helped |
| add fine-tuning on monocular observations (ablation) | Vid2Avatar-Pro · MonoPerfCap | LPIPS (×100): 2.29 -> 1.37 | Helped |
| add average-scale skeleton normalization (ablation) | Vid2Avatar-Pro · MonoPerfCap | PSNR: 31.03 -> 31.97 | Helped |
| add average-scale skeleton normalization (ablation) | Vid2Avatar-Pro · MonoPerfCap | LPIPS (×100): 1.51 -> 1.37 | Helped |
| increase universal-prior training identities from 4 to 16 (ablation) | Vid2Avatar-Pro · MonoPerfCap | LPIPS (×100): 1.53 -> 1.51 | Helped |
| increase universal-prior training identities from 16 to 128 (ablation) | Vid2Avatar-Pro · MonoPerfCap | LPIPS (×100): 1.51 -> 1.45 | Helped |
Limitations
- Universal prior training data lacks diverse facial expressions, so animatable faces are not supported.
- Loose garments are not well covered by the training data, so loose clothing is not supported.
- Assumes standard lighting conditions and may underperform under extreme lighting variations.
Experimental setup
Datasets: in-house multi-view human performance capture (1,000 subjects) NeuMan MonoPerfCap
Benchmarks: NeuMan MonoPerfCap
Metrics: PSNR SSIM LPIPS
Baselines: HumanNeRF InstantAvatar NeuMan Vid2Avatar 3DGS-Avatar GaussianAvatar ExAvatar
Builds on: 3D Gaussian Splatting Vid2Avatar Li et al. front and back position maps SMPL-X DiT latent diffusion
Closest papers by meaning
- FRESA: Feedforward Reconstruction of Personalized Skinned Avatars from Few Images (CVPR 2025 · similarity 0.90)
- WildAvatar: Learning In-the-wild 3D Avatars from the Web (CVPR 2025 · similarity 0.89)
- RAM-Avatar: Real-time Photo-Realistic Avatar from Monocular Videos with Full-body Control (CVPR 2024 · similarity 0.89)
- IntrinsicAvatar: Physically Based Inverse Rendering of Dynamic Humans from Monocular Videos via Explicit Ray Tracing (CVPR 2024 · similarity 0.89)
- Synthetic Prior for Few-Shot Drivable Head Avatar Inversion (CVPR 2025 · similarity 0.89)
- Expressive Gaussian Human Avatars from Monocular RGB Video (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.