Research map / 3D vision / Gaussian splatting
OccFusion: Rendering Occluded Humans with Generative Diffusion Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
OccFusion renders occluded humans from monocular video by training 3D Gaussians (built on GauHuman) with SDS supervision from Stable Diffusion 1.5 + ControlNet and an in-context inpainting refinement stage.
Problem
Rendering humans from monocular videos when parts of the body are persistently occluded, leading to incomplete geometry and artifacts in existing human rendering methods that assume full visibility.
Approach
Combines efficient 3D Gaussian splatting (an occlusion-tuned GauHuman variant, OccGauHuman) with a pretrained 2D diffusion prior applied three ways: inpainting occluded human masks, SDS on rendered occupancy maps in posed and canonical space, and in-context inpainting of RGB references for refinement.
Key results
- On ZJU-MoCap, OccFusion reaches PSNR 23.96, SSIM 0.9548, LPIPS 32.34 (x1000), versus OccGauHuman 22.71 / 0.9492 / 54.60 and GauHuman 21.55 / 0.9430 / 55.88.
- On OcMotion (visible-pixel metrics), OccFusion attains SSIM 0.8875 and LPIPS 82.42, best on SSIM and LPIPS, while OccGauHuman has higher PSNR (18.85 vs 18.28).
- The full pipeline trains in about 10 minutes on a single TITAN RTX GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply occlusion-specific GauHuman tweaks (visible-pixel training, higher mask-loss weight, and disabled densification/pruning) (ablation) | GauHuman · ZJU-MoCap | PSNR: 21.55 -> 22.54 | Helped |
| Add generated human masks from the Init Stage (ablation) | OccGauHuman · ZJU-MoCap | PSNR: 22.54 -> 23.52 | Helped |
| Add posed-space SDS (ablation) | OccGauHuman + Init Stage · ZJU-MoCap | LPIPS (×1000): 52.35 -> 55.47 | Hurt |
| Add canonical-space SDS (ablation) | OccGauHuman + Posed space SDS · ZJU-MoCap | LPIPS (×1000): 55.47 -> 55.35 | Helped |
| Add the Refinement Stage (ablation) | OccGauHuman + Canonical space SDS · ZJU-MoCap | LPIPS (×1000): 55.35 -> 32.34 | Helped |
| Use in-context inpainting references during refinement | OccFusion Refinement Stage · ZJU-MoCap | rendering quality in occluded areas | Helped |
| Simplify pose conditioning by ignoring self-occluded joints | OccFusion initialization · challenging poses | generation quality | Helped |
Surprising findings
- Applying SDS to rendered occupancy masks rather than RGB images, and the Initialization/SDS stages alone do not recover missing appearance; only the in-context inpainting Refinement stage substantially improves LPIPS (55.35 to 32.34).
- Off-the-shelf single-image generative model InstantMesh fails to recover a complete human geometry from the least occluded frame.
Limitations
- Generated inpaintings are sometimes incoherent across frames, which can hurt rendering model training.
- Conditioning Stable Diffusion on 2D poses is weak; generated poses do not always match the condition.
- Relies on accurate ground-truth or externally estimated poses and segmentation masks.
Experimental setup
Datasets: ZJU-MoCap OcMotion
Benchmarks: ZJU-MoCap OcMotion
Metrics: PSNR SSIM LPIPS training time
Baselines: HumanNeRF 3DGS-Avatar GauHuman OccNeRF OccGaussian Wild2Avatar OccGauHuman InstantMesh
Builds on: GauHuman 3D Gaussian splatting Score Distillation Sampling (DreamFusion) ControlNet SMPL
Closest papers by meaning
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.88)
- HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting (CVPR 2024 · similarity 0.87)
- Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpses (CVPR 2024 · similarity 0.87)
- GaussianIP: Identity-Preserving Realistic 3D Human Generation via Human-Centric Diffusion Prior (CVPR 2025 · similarity 0.87)
- Human Gaussian Splatting: Real-time Rendering of Animatable Avatars (CVPR 2024 · similarity 0.87)
- DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recovery (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.