Research map / 3D vision / Face and shape
Stratified Avatar Generation from Sparse Observations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAGE, a stratified latent-diffusion method that reconstructs upper-body motion from HMD sparse signals first, then lower-body motion conditioned on it, using disentangled VQ-VAEs on AMASS.
Problem
Reconstructing full-body 3D avatars, especially the lower body, from sparse HMD observations (head and two hands) where no direct lower-body tracking signals exist.
Approach
Exploits the SMPL kinematic tree, where upper and lower body share only the root joint, to split avatar reconstruction into a two-stage pipeline: upper-body latent diffusion from sparse inputs, then lower-body latent diffusion conditioned on the predicted upper-body latent, with a full-body decoder trained jointly.
Key results
- Under setting S1, SAGE reaches MPJPE 3.28 and Lower PE 6.01 versus AvatarJLM's 3.35 and 6.14, the best prior result.
- Under setting S3, SAGE achieves MPJPE 2.95 and Lower PE 5.37, outperforming AGRoL (offline) at 3.80 and 6.90.
- Stratified diffusion lowers lower-body position error from 6.73 (parallel diffusion) to 6.46 and jitter from 14.71 to 10.83 in ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use disentangled upper- and lower-body codebooks instead of a unified full-body codebook (ablation) | SAGE · AMASS S1 | MPJPE: 3.62 -> 3.28 | Helped |
| add the full-body decoder (ablation) | SAGE · AMASS S1 | MPJPE: 3.69 -> 3.28 | Helped |
| add the temporal Refiner (ablation) | SAGE · AMASS S1 | MPJPE: 3.26 -> 3.28 | No clear effect |
| disentangle the body into five root-to-leaf segments rather than upper and lower body (ablation) | SAGE · AMASS S1 | MPJPE: 3.28 -> 3.48 | Hurt |
| condition lower-body generation on predicted upper-body latents rather than sparse observations alone (ablation) | SAGE · AMASS S1 | Lower PE: 6.73 -> 6.46 | Helped |
| condition lower-body generation on predicted upper-body latents rather than sparse observations alone (ablation) | SAGE · AMASS S1 | Jitter: 14.71 -> 10.83 | Helped |
| use SAGE instead of AvatarJLM | HMD avatar reconstruction · AMASS S1 | MPJRE: 2.90 -> 2.53 | Helped |
| use SAGE instead of AGRoL (Offline) | HMD avatar reconstruction · AMASS S1 | MPJVE: 18.59 -> 20.62 | Hurt |
| use SAGE instead of AvatarJLM | HMD avatar reconstruction · AMASS S1 with root input | MPJPE: 2.09 -> 1.88 | Helped |
| use SAGE instead of AvatarJLM | HMD avatar reconstruction · AMASS S2 | MPJPE: 4.93 -> 5.86 | Hurt |
| use SAGE instead of AvatarJLM | HMD avatar reconstruction · AMASS S3 | MPJPE: 3.39 -> 2.95 | Helped |
Surprising findings
- Further splitting the body into five root-to-leaf paths hurt performance, since it disrupted natural upper/lower-body joint connections.
Limitations
- Fails on external force-induced movements such as being pushed or dragged.
- Struggles with unconventional poses not well represented in training data.
- The S2 test split is small (about 1% of training), limiting evaluation of scalability.
Experimental setup
Datasets: AMASS CMU BMLrub HDM05 Transition HumanEva
Benchmarks: AMASS
Metrics: MPJRE MPJPE MPJVE Jitter Hand PE Upper PE Lower PE Root PE
Baselines: Final IK LoBSTr VAE-HMD AvatarPoser AvatarJLM AGRoL CoolMoves Humor FLAG
Builds on: VQ-VAE latent diffusion model AvatarPoser AvatarJLM AGRoL
Closest papers by meaning
- Vid2Avatar-Pro: Authentic Avatar from Videos in the Wild via Universal Prior (CVPR 2025 · similarity 0.87)
- GeoAvatar: Geometrically-Consistent Multi-Person Avatar Reconstruction from Sparse Multi-View Videos (CVPR 2025 · similarity 0.86)
- FRESA: Feedforward Reconstruction of Personalized Skinned Avatars from Few Images (CVPR 2025 · similarity 0.86)
- RAM-Avatar: Real-time Photo-Realistic Avatar from Monocular Videos with Full-body Control (CVPR 2024 · similarity 0.86)
- Real-Time Simulated Avatar from Head-Mounted Sensors (CVPR 2024 · similarity 0.86)
- Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.