Research map / 3D vision / Face and shape
Real-time Free-view Human Rendering from Sparse-view RGB Videos using Double Unprojected Textures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Double Unprojected Textures (DUT), which disentangles coarse image-conditioned template deformation (GeoNet) from Gaussian appearance prediction (GauNet) via two texture unprojections, enabling real-time 4K sparse-view human rendering.
Problem
Real-time free-view human rendering from sparse-view (1-4) RGB videos, where prior texture-space methods either jointly learn geometry and appearance from imperfect unprojections or ignore image cues for geometry, hurting visual quality and robustness to unseen poses.
Approach
Unproject images onto the posed template once to predict coarse image-conditioned vertex offsets with a UNet, then unproject again onto the deformed template to get a less distorted texture for predicting 3D Gaussian parameters in texel space, plus a Gaussian scale refinement based on LBS edge scaling ratios.
Key results
- On S3 at 4K, Ours-Large reaches 30.03 PSNR / 0.872 SSIM vs. 29.23 for GHG and 29.37 for DVA.
- On S2618 at full resolution, Ours achieves 29.38 PSNR vs. 29.28 for DVA and 28.50 for GHG; Ours-Large reaches 29.69.
- Runtime is reported as three times faster than HoloChar, with real-time 4K rendering on a single RTX3090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use image-conditioned template deformation with double unprojection and Gaussian appearance prediction | DUT · S3 - 1K | PSNR: 31.4086 -> 32.8897 | Helped |
| Use image-conditioned template deformation with double unprojection and Gaussian appearance prediction | DUT · S22 - 1K | PSNR: 33.7842 -> 34.1119 | Helped |
| Use image-conditioned template deformation with double unprojection and Gaussian appearance prediction | DUT · S2618 - Half Res | LPIPS: 0.0774 -> 0.0747 | Helped |
| Add RGB image features to GeoNet (ablation) | DUT template deformation · Template deformation | CD: 1.150 -> 1.070 | Helped |
| Add normal features to GeoNet (ablation) | DUT template deformation · Template deformation | CD: 1.117 -> 1.070 | Helped |
| Add template deformation rather than using LBS alone (ablation) | DUT template deformation · Template deformation | CD: 2.117 -> 1.070 | Helped |
| Increase the number of input views from 1 to 2 (ablation) | DUT appearance module · S3 - 4K | PSNR: 27.9061 -> 29.3016 | Helped |
| Use 2 input views rather than 0 (ablation) | DUT appearance module · S3 - 4K | PSNR: 25.1546 -> 29.3016 | Helped |
| Add double unprojection (ablation) | DUT appearance module · S3 - 4K | PSNR: 29.4185 -> 29.8369 | Helped |
| Add Gaussian scale refinement (ablation) | DUT appearance module · S3 - 4K | PSNR: 29.8102 -> 29.8369 | No clear effect |
| Increase Gaussian texture resolution from 256 to 512 (ablation) | DUT appearance module · S3 - 4K | PSNR: 29.8369 -> 30.0311 | Helped |
| Use geometry from GeoNet without RGB features (ablation) | DUT appearance module · S3 - 4K | PSNR: 29.7097 -> 29.8369 | Hurt |
Surprising findings
- Removing the second (double) unprojection only modestly degrades PSNR (29.42 vs 29.84 at 4K), suggesting appearance gains are partly from other components.
- The HoloChar baseline achieves lower PSNR than the full model on OOD motions (30.22 vs 32.64), while motion-only conditioning was reported as less robust.
Limitations
- Cannot maintain finger rigidity; hand geometry deforms unrealistically.
- Template-based approach struggles with topological changes such as unzipping a jacket.
- Requires scans or point clouds (NeuS2 reconstructions) per frame for deformation supervision.
Experimental setup
Datasets: DynaCap ASH THUman4.0
Benchmarks: DynaCap S3 ASH S22 THUman4.0 S2618
Metrics: PSNR SSIM LPIPS Chamfer distance surface regularity self-intersection runtime
Baselines: ENeRF DVA HoloChar GHG DDC LBS
Builds on: 3D Gaussian Splatting DDC ASH HoloChar GHG
Closest papers by meaning
- Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras (CVPR 2024 · similarity 0.89)
- Human Gaussian Splatting: Real-time Rendering of Animatable Avatars (CVPR 2024 · similarity 0.87)
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.86)
- Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (ICLR 2024 · similarity 0.86)
- RoGSplat: Learning Robust Generalizable Human Gaussian Splatting from Sparse Multi-View Images (CVPR 2025 · similarity 0.86)
- RigPAPR: Rig-Based Animation of Static Neural Point Clouds from a Single-View Video (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.