Research map / 3D vision / Face and shape
Pose Modulated Avatars from Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a pose-modulated frequency NeRF for human avatars, where a GNN over skeleton joints drives per-query sine-activation frequency coefficients, evaluated on Human3.6M and MonoPerfCap.
Problem
Learning pose-dependent cloth and skin deformation and fine texture detail for NeRF-based human avatars from sparse or monocular video, where fixed positional encodings and implicit mappings yield either oversmoothed details or noisy artifacts.
Approach
A two-branch network in which a GNN extracts per-bone features from skeleton pose and a two-stage learnable window function aggregates them into pose- and point-dependent frequency coefficients that modulate Sine-activated layers of a NeRF, making frequency explicitly depend on pose context.
Key results
- Novel-view synthesis on Human3.6M averages PSNR 25.06 / SSIM 0.921 / LPIPS 0.110, beating DANBO (24.69 / 0.917 / 0.116) and TAVA (24.72 / 0.919 / 0.124).
- Novel-pose rendering on Human3.6M averages PSNR 24.15 / SSIM 0.906 / LPIPS 0.124, versus DANBO at 23.74 / 0.901 / 0.131.
- Ablation on MonoPerfCap novel pose shows the full model at PSNR 25.12 versus 24.75 for onlyGNN and 20.98 for no window.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pose-guided frequency modulation instead of the DANBO approach | Human avatar NeRF · Human3.6M test set, novel view (Avg) | PSNR: 24.69 -> 25.06 | Helped |
| Use pose-guided frequency modulation instead of the DANBO approach | Human avatar NeRF · Human3.6M test set, novel pose (Avg) | PSNR: 23.74 -> 24.15 | Helped |
| Add the GNN feature branch to frequency modulation (ablation) | Human avatar NeRF · Human3.6M S9, novel pose | PSNR: 24.82 -> 25.12 | Helped |
| Add the frequency-modulation branch to GNN features (ablation) | Human avatar NeRF · Human3.6M S9, novel pose | PSNR: 24.75 -> 25.12 | Helped |
| Add the feature window function alongside the spatial window (ablation) | Human avatar NeRF · Human3.6M S9, novel pose | PSNR: 25.01 -> 25.12 | Helped |
| Add the spatial window function alongside the feature window (ablation) | Human avatar NeRF · Human3.6M S9, novel pose | PSNR: 24.72 -> 25.12 | Helped |
| Add the two-stage window function (ablation) | Human avatar NeRF · Human3.6M S9, novel pose | PSNR: 20.98 -> 25.12 | Helped |
Limitations
- Person-specific, requires individual training per subject
- Relies on accurate camera parameters and is not real-time
- Blurry artifacts under extreme out-of-distribution poses
Experimental setup
Datasets: Human3.6M MonoPerfCap ZJU-Mocap AIST
Benchmarks: Human3.6M MonoPerfCap AIST
Metrics: PSNR SSIM LPIPS
Baselines: NeuralBody Anim-NeRF A-NeRF TAVA DANBO ARAH
Builds on: NeRF DANBO pi-GAN SIREN Anim-NeRF
Closest papers by meaning
- Structured Local Radiance Fields for Human Avatar Modeling (CVPR 2022 · similarity 0.90)
- REACTO: Reconstructing Articulated Objects from a Single Video (CVPR 2024 · similarity 0.88)
- VINECS: Video-based Neural Character Skinning (CVPR 2024 · similarity 0.88)
- Relightable and Animatable Neural Avatars from Videos (AAAI 2024 · similarity 0.88)
- 3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting (CVPR 2024 · similarity 0.88)
- Human Gaussian Splatting: Real-time Rendering of Animatable Avatars (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.