Research map / 3D vision / Face and shape
Relightable and Animatable Neural Avatars from Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds relightable, animatable neural human avatars from sparse videos under unknown lighting using an invertible deformation field for geometry and part-wise light visibility networks for pose-aware shadows.
Problem
Creating relightable and animatable 3D clothed human avatars from sparse videos under unknown illumination, requiring disentanglement of geometry, material, and lighting under complex pose-dependent self-shadowing.
Approach
An invertible neural deformation field (Real-NVP-style coupling blocks conditioned on pose and skinning weights) gives bidirectional canonical-observation mapping; skinning weights come from a deformed explicit body mesh; light visibility is predicted by 15 part-wise pose-aware networks whose outputs are multiplied.
Key results
- On synthetic data, relighting on novel poses reaches PSNR 19.86 vs 16.75 for Relighting4D, and geometry Chamfer distance averages 0.431 vs 0.541 for ARAH.
- The invertible deformation network has 1.66% relative cycle-consistency error versus 9.45% for a single-direction MLP.
- Rendering a 512x512 image takes about 40 s; training takes about 2.5 days on a single RTX 3090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add mesh-based inverse skinning using the extracted body mesh rather than SMPL mesh weights (ablation) | Ours geometry reconstruction · Synthetic geometry | P2S: 0.237 -> 0.182 | Helped |
| add mesh-based inverse skinning using the extracted body mesh rather than SMPL mesh weights (ablation) | Ours geometry reconstruction · Synthetic geometry | CD: 0.485 -> 0.431 | Helped |
| add skinning weights W as a deformation-network condition (ablation) | Ours geometry reconstruction · Synthetic geometry | P2S: 0.248 -> 0.182 | Helped |
| add skinning weights W as a deformation-network condition (ablation) | Ours geometry reconstruction · Synthetic geometry | CD: 0.507 -> 0.431 | Helped |
| add light visibility modeling (ablation) | Ours relighting · Synthetic relighting (training poses) | PSNR: 23.7758 -> 25.3477 | Helped |
| add light visibility modeling (ablation) | Ours relighting · Synthetic relighting (novel poses) | PSNR: 18.9768 -> 19.8622 | Helped |
| use part-wise visibility networks instead of a single visibility network (ablation) | Ours relighting · Synthetic relighting (training poses) | PSNR: 24.7064 -> 25.3477 | Helped |
| use part-wise visibility networks instead of a single visibility network (ablation) | Ours relighting · Synthetic relighting (novel poses) | PSNR: 19.7119 -> 19.8622 | Helped |
| use part-wise visibility networks instead of a single visibility network (ablation) | Ours albedo estimation · Synthetic albedo map | PSNR: 25.2150 -> 25.1666 | No clear effect |
| replace Relighting4D with Ours | Synthetic relighting · Synthetic relighting (novel poses) | PSNR: 16.7475 -> 19.8622 | Helped |
Surprising findings
- Removing the part-wise design yields nearly equal albedo quality and only modest relighting drops, while removing visibility entirely loses self-occlusion shadows.
- Against non-relightable methods on Human3.6M novel-view and novel-pose synthesis, the method is only comparable to state-of-the-art rather than clearly better.
Limitations
- Slow inference (about 40 s per image) and long training (about 2.5 days on one GPU).
- Cannot model pose-dependent wrinkles for loose clothing or global illumination effects.
- Only models body motion, not face and hands.
Experimental setup
Datasets: ZJU-MoCap Human3.6M DeepCap PeopleSnapshot AIST++ Mixamo
Benchmarks: Human3.6M ZJU-MoCap DeepCap PeopleSnapshot synthetic Mixamo relighting dataset
Metrics: PSNR SSIM LPIPS P2S Chamfer Distance
Baselines: Relighting4D ARAH Peng et al. NeuralBody Anim-NeRF A-NeRF DANBO TAVA NPC MonoHuman
Builds on: NeRF VolSDF Real-NVP SMPL Relighting4D
Closest papers by meaning
- Relightable and Animatable Neural Avatar from Sparse-View Video (CVPR 2024 · similarity 0.90)
- BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading (NeurIPS 2025 · similarity 0.90)
- Artist-Friendly Relightable and Animatable Neural Heads (CVPR 2024 · similarity 0.89)
- NECA: Neural Customizable Human Avatar (CVPR 2024 · similarity 0.89)
- 3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting (CVPR 2024 · similarity 0.89)
- IntrinsicAvatar: Physically Based Inverse Rendering of Dynamic Humans from Monocular Videos via Explicit Ray Tracing (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.