Research map / 3D vision / Face and shape
Relightable and Animatable Neural Avatar from Sparse-View Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a relightable and animatable neural human avatar from sparse-view or monocular video under unknown illumination, using a Hierarchical Distance Query so sphere tracing and distance-field soft shadows work on deformed SDFs.
Problem
Creating relightable and animatable neural human avatars from sparse-view or monocular videos under unknown lighting, where sphere tracing on deformed SDFs is inaccurate and costly for visibility computation.
Approach
A Hierarchical Distance Query blends a coarse world-space KNN distance to the posed SMPL-H mesh with a fine canonical neural SDF distance, giving an approximate world-space SDF for sphere tracing; this is combined with distance field soft shadows (DFSS) on the deformed SDF for single-sample soft visibility.
Key results
- On SyntheticHuman++ sparse-view, the method reaches relighting PSNR 20.20 and diffuse albedo PSNR 22.69, versus 15.22 and 22.13 for Relighting4D*.
- Hierarchical Distance Query gives about 10x speed-up over volume rendering for surface intersection (5s vs 60s per 512x512 image).
- Training takes 20 hours on an RTX 3090 and rendering a 512x512 image takes 5s, which the authors identify as a limitation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add fine canonical distance to coarse GS-KNN distance in HDQ (ablation) | Hierarchical Distance Query · SyntheticHuman++ sparse-view, jody | Relighting PSNR: 19.56 -> 21.57 | Helped |
| add coarse world-space distance to HDQ (ablation) | Hierarchical Distance Query · SyntheticHuman++ sparse-view, jody | Relighting PSNR: 20.69 -> 21.57 | Helped |
| use sphere tracing instead of volume rendering of 128 samples per ray (ablation) | Ours · SyntheticHuman++ sparse-view, jody | Relighting PSNR: 21.36 -> 21.57 | Helped |
| add the light visibility term (ablation) | Ours · SyntheticHuman++ sparse-view, jody | Relighting PSNR: 21.00 -> 21.57 | Helped |
| use soft visibility instead of hard visibility (ablation) | Ours · SyntheticHuman++ sparse-view, jody | Relighting PSNR: 21.19 -> 21.57 | Helped |
| set cut-offs to Td = 2.0 and Tvis_d = 0.5 (ablation) | Ours · SyntheticHuman++ sparse-view, jody | Relighting PSNR: 21.57 -> 21.11 | No clear effect |
| set cut-offs to Td = 1.0 and Tvis_d = 0.25 (ablation) | Ours · SyntheticHuman++ sparse-view, jody | Relighting PSNR: 21.57 -> 21.11 | No clear effect |
| set cut-offs to Td = 0.5 and Tvis_d = 0.125 (ablation) | Ours · SyntheticHuman++ sparse-view, jody | Relighting PSNR: 21.57 -> 21.10 | No clear effect |
| set cut-offs to Td = 0.1 and Tvis_d = 0.025 (ablation) | Ours · SyntheticHuman++ sparse-view, jody | Relighting PSNR: 21.57 -> 21.08 | No clear effect |
| set cut-offs to Td = 0.05 and Tvis_d = 0.0125 (ablation) | Ours · SyntheticHuman++ sparse-view, jody | Relighting PSNR: 21.57 -> 21.04 | No clear effect |
| set cut-offs to Td = 0.01 and Tvis_d = 0.0025 (ablation) | Ours · SyntheticHuman++ sparse-view, jody | Relighting PSNR: 21.57 -> 19.50 | Hurt |
Surprising findings
- Naively sphere tracing on the canonical SDF or on the coarse GS-KNN world distance alone produces incorrect surface intersections and visibility, motivating the blending.
Limitations
- Long training time of about 20 hours per avatar.
- Rendering takes about 5 seconds per 512x512 image, not real-time.
- Relies on provided human poses and foreground masks.
Experimental setup
Datasets: MobileStage SyntheticHuman++ SyntheticHuman
Benchmarks: MobileStage SyntheticHuman++
Metrics: PSNR SSIM LPIPS normal degree error
Baselines: NeRFactor Relighting4D AniSDF
Builds on: AniSDF NeRFactor L-Tracing Distance Field Soft Shadow Neural Body
Closest papers by meaning
- Relightable and Animatable Neural Avatars from Videos (AAAI 2024 · similarity 0.90)
- Artist-Friendly Relightable and Animatable Neural Heads (CVPR 2024 · similarity 0.88)
- 3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting (CVPR 2024 · similarity 0.88)
- IntrinsicAvatar: Physically Based Inverse Rendering of Dynamic Humans from Monocular Videos via Explicit Ray Tracing (CVPR 2024 · similarity 0.88)
- NECA: Neural Customizable Human Avatar (CVPR 2024 · similarity 0.87)
- BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.