Research map / 3D vision / Face and shape
Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Holoported Characters renders real-time, 4K free-viewpoint videos of a person from only four sparse RGB camera views plus 3D skeletal pose, by combining a skeleton-driven mesh model, projective texture encoding, and a super-resolution network.
Problem
Real-time, high-resolution free-viewpoint rendering of humans in loose clothing with fine dynamic details (wrinkles, faces, hands) from sparse multi-view input and skeletal pose, which prior methods do not achieve jointly.
Approach
Three-stage pipeline: a DDC-based deformable mesh refined with Chamfer supervision from neural surface reconstructions and hand-vertex constraints; projective texturing of the four input views into texture space; and a TexFeatNet plus a shallow image-space super-resolution CNN (SRNet) that produces 4K renderings in real time.
Key results
- Runs in real time at 4112x3008 on dual NVIDIA A100 GPUs (GPU 1 at 22 FPS for character model and texturing, GPU 2 at about 25 FPS for TexFeatNet and SRNet).
- On DynaCap subject S2 (loose clothing) novel view, LPIPS improves to 21.86 (x1000) and FID to 5.95 vs DDC's 47.23 and 36.33; LPIPS is 12.98 vs DDC's 20.07 on S1.
- Performance on par with HDHumans (non-real-time) while outperforming all real-time methods in LPIPS and FID.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add partial texture input to TexFeatNet (ablation) | Holoported Characters · DynaCap S1 novel pose | PSNR: 26.44 -> 30.24 | Helped |
| add partial texture input to TexFeatNet (ablation) | Holoported Characters · DynaCap S1 novel pose | LPIPS (×1000): 44.34 -> 23.74 | Helped |
| add partial texture input to TexFeatNet (ablation) | Holoported Characters · DynaCap S1 novel pose | FID: 66.13 -> 11.62 | Helped |
| add TexFeatNet output features (ablation) | Holoported Characters · DynaCap S1 novel pose | PSNR: 29.89 -> 30.24 | Helped |
| add TexFeatNet output features (ablation) | Holoported Characters · DynaCap S1 novel pose | LPIPS (×1000): 25.52 -> 23.74 | Helped |
| add TexFeatNet output features (ablation) | Holoported Characters · DynaCap S1 novel pose | FID: 14.02 -> 11.62 | Helped |
| add Chamfer loss surface supervision (ablation) | Holoported Characters · DynaCap S1 novel pose | PSNR: 30.04 -> 30.24 | Helped |
| add super-resolution module (ablation) | Holoported Characters · DynaCap S1 novel pose | PSNR: 29.76 -> 30.24 | Helped |
| train on 4K images (ablation) | Holoported Characters · DynaCap S1 novel pose, 1K evaluation | LPIPS (×1000): 24.95 -> 23.74 | Helped |
| train on 4K images (ablation) | Holoported Characters · DynaCap S1 novel pose, 4K evaluation | LPIPS (×1000): 33.35 -> 32.4 | Helped |
| use Holoported Characters instead of DDC | DynaCap subjects S1 and S2 · novel view and novel pose synthesis | PSNR: 32.96 (28.05) -> 33.93 (30.24); 27.92 (25.92) -> 31.45 (28.03) | Helped |
| use Holoported Characters instead of HDHumans | DynaCap subjects S1 and S2 · novel view and novel pose synthesis | PSNR: 31.00 (27.69) -> 33.93 (30.24); 28.98 (26.32) -> 31.45 (28.03) | Mixed |
Surprising findings
- Training on 4K images improves metrics even when evaluated at 1K resolution, especially LPIPS and FID.
Limitations
- Cannot handle topological changes such as opening a jacket.
- Requires dense multi-view video and a rigged static scan for training.
- Requires markerless motion capture input for skeletal pose.
Experimental setup
Datasets: DynaCap
Benchmarks: DynaCap
Metrics: PSNR LPIPS FID
Baselines: DDC HDHumans ENeRF DVA Neural Image-based Avatars UV Volumes
Builds on: DDC (Habermann et al.) Embedded Graph deformation Dual Quaternion Skinning Neural Surface Reconstruction
Closest papers by meaning
- Real-time Free-view Human Rendering from Sparse-view RGB Videos using Double Unprojected Textures (CVPR 2025 · similarity 0.89)
- Human Gaussian Splatting: Real-time Rendering of Animatable Avatars (CVPR 2024 · similarity 0.86)
- IntrinsicAvatar: Physically Based Inverse Rendering of Dynamic Humans from Monocular Videos via Explicit Ray Tracing (CVPR 2024 · similarity 0.86)
- Link to the Past: Temporal Propagation for Fast 3D Human Reconstruction from Monocular Video (CVPR 2025 · similarity 0.85)
- ConTex-Human: Free-View Rendering of Human from a Single Image with Texture-Consistent Synthesis (CVPR 2024 · similarity 0.85)
- GBC-Splat: Generalizable Gaussian-Based Clothed Human Digitalization under Sparse RGB Cameras (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.