Research map / 3D vision / Face and shape
RAM-Avatar: Real-time Photo-Realistic Avatar from Monocular Videos with Full-body Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAM-Avatar, a real-time neural-texture image-to-image framework that learns photo-realistic full-body (body, face, hands) avatars from monocular videos, using a sparse dual attention module, StyleUnet with temporal discriminator, and a motion distribution align module.
Problem
Learning animatable human avatars from monocular video that render photo-realistically in real time with full-body control (face and hands), and remain robust to out-of-distribution poses.
Approach
Combines FaceVerse and MANO statistical templates for facial and hand control, a sparsely computed (criss-cross) spatial-temporal dual attention over neural-texture body features, a lightweight StyleUnet with a temporal-aware discriminator, and a VAE-based Motion Distribution Align module that maps out-of-distribution poses into the avatar's training pose distribution at test time.
Key results
- On novel in-distribution poses, RAM-Avatar reaches SSIM 0.943, LPIPS 0.036, FID 12.79, beating HFMT (FID 14.48) and InstantAvatar (FID 44.48).
- On novel large-rotation poses, it achieves SSIM 0.961 and LPIPS 0.020, best among compared methods on SSIM/LPIPS, though FID (32.09) trails InstantAvatar (26.97).
- Out-of-distribution pose animation: ANR and InstantAvatar fail to render reasonable results while RAM-Avatar adjusts motion range without collapse (qualitative).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RAM-Avatar instead of the strongest baseline for each metric | human avatar animation · Novel in-distribution pose animation | SSIM: 0.918 -> 0.943 | Helped |
| Use RAM-Avatar instead of the strongest baseline for each metric | human avatar animation · Novel in-distribution pose animation | LPIPS: 0.053 -> 0.036 | Helped |
| Use RAM-Avatar instead of the strongest baseline for each metric | human avatar animation · Novel in-distribution pose animation | tLPIPS: 0.405 -> 0.353 | Helped |
| Use RAM-Avatar instead of the strongest baseline for each metric | human avatar animation · Novel in-distribution pose animation | FID: 14.48 -> 12.79 | Helped |
| Use RAM-Avatar instead of the strongest baseline for each metric | human avatar animation · Novel large rotation pose animation | SSIM: 0.940 -> 0.961 | Helped |
| Use RAM-Avatar instead of the strongest baseline for each metric | human avatar animation · Novel large rotation pose animation | LPIPS: 0.039 -> 0.020 | Helped |
| Use RAM-Avatar instead of the strongest baseline for each metric | human avatar animation · Novel large rotation pose animation | tLPIPS: 0.174 -> 0.155 | Helped |
| Use RAM-Avatar instead of the strongest baseline for each metric | human avatar animation · Novel large rotation pose animation | FID: 26.97 -> 32.09 | Hurt |
| Add the spatial part of the dual attention module (ablation) | RAM-Avatar · Novel in-distribution pose animation | FID: 13.23 -> 12.79 | Helped |
| Add the temporal part of the dual attention module (ablation) | RAM-Avatar · Novel in-distribution pose animation | tLPIPS: 0.380 -> 0.353 | Helped |
| Add hand modeling (ablation) | RAM-Avatar · Novel in-distribution pose animation | image quality | Helped |
| Add face modeling (ablation) | RAM-Avatar · Novel in-distribution pose animation | image quality | Helped |
Surprising findings
- InstantAvatar achieves good results with 300 training frames but produces blurry results with larger datasets.
Limitations
- Image-to-image translation is constrained by training data; rotation poses unseen in training yield blurry results.
- Parametric template tracking does not accurately describe body poses or facial expressions, so full-body control is imperfect.
Experimental setup
Datasets: HFshirt People-Snapshot female-4-sport Male1 (self-collected) AMASS-style motion capture datasets
Benchmarks: HFshirt People-Snapshot female-4-sport Male1
Metrics: SSIM LPIPS tLPIPS FID
Baselines: ANR HFMT InstantAvatar
Builds on: ANR Neural Texture Style-Unet criss-cross attention dual attention network VPoser
Closest papers by meaning
- Vid2Avatar-Pro: Authentic Avatar from Videos in the Wild via Universal Prior (CVPR 2025 · similarity 0.89)
- Expressive Gaussian Human Avatars from Monocular RGB Video (NeurIPS 2024 · similarity 0.88)
- NECA: Neural Customizable Human Avatar (CVPR 2024 · similarity 0.87)
- Real-time High-fidelity Gaussian Human Avatars with Position-based Interpolation of Spatially Distributed MLPs (CVPR 2025 · similarity 0.87)
- HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation (NeurIPS 2024 · similarity 0.87)
- CtrlAvatar: Controllable Avatars Generation via Disentangled Invertible Networks (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.