Research map / 3D vision / 3D generation
Single-view Image to Novel-view Generation for Hand-Object Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Synthesizes novel-view hand-object interaction images from a single RGB image by fine-tuning Zero123 with hand depth/skeleton conditioning and lifting the outputs to 3D Gaussian splatting with contact-guided local hand embedding.
Problem
Generating photo-realistic novel-view renderings of hand-object interactions from a single RGB image, where prior work yields texture-less meshes or needs multi-view videos, and occlusion by the hand is hard to handle.
Approach
Two-stage pipeline: a Zero123 diffusion prior fine-tuned with a global hand embedding (ControlNet-style hand depth plus a skeleton-branch UNet from MANO-derived maps) for viewpoint and hand control, followed by 3D Gaussian splatting trained with SDS and a local contact field that pulls occluded object Gaussians toward the hand surface.
Key results
- On HO3D novel-view synthesis, HO123 reaches 31.26 PSNR, 0.97 SSIM, 0.037 LPIPS, versus 26.01 PSNR for MOHO and 20.54 for HandNeRF.
- On HO3D shape reconstruction, HO123 achieves Chamfer Distance 0.45 mm and F-10 0.61, beating MOHO (0.91, 0.50) and DDF-HO (0.55, 0.42).
- Ablation on HO3D shows removing the local hand embedding raises CD from 0.45 to 2.46 when combined with global embeddings... (see Table 3), with full model PSNR 31.26.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HO123 instead of MOHO | novel-view synthesis · HO3D | PSNR: 26.01 -> 31.26 | Helped |
| use HO123 instead of MOHO | novel-view synthesis · DexYCB | PSNR: 35.80 -> 36.40 | Helped |
| use HO123 instead of DDF-HO | single-view shape reconstruction · HO3D | CD (mm): 0.55 -> 0.45 | Helped |
| use HO123 instead of MOHO | single-view shape reconstruction · DexYCB | F-10: 0.81 -> 0.87 | Helped |
| add global pose embedding (ablation) | HO123 · HO3D | PSNR: 18.23 -> 27.35 | Helped |
| add global shape embedding (ablation) | HO123 · HO3D | PSNR: 18.23 -> 28.36 | Helped |
| add local hand embedding (ablation) | HO123 with GPE and GSE · HO3D | CD (mm): 2.46 -> 0.45 | Helped |
| add global pose and shape embeddings together without local hand embedding (ablation) | HO123 · HO3D | CD (mm): 0.45 -> 2.46 | Hurt |
Surprising findings
- Directly fine-tuning Zero123 on hand-object data gives little improvement (Zero123* 20.15 PSNR on HO3D), suggesting the diffusion model needs explicit hand conditioning rather than fine-tuning alone.
Limitations
- Focuses on static hand-object interactions; not animatable.
- Relies on an off-the-shelf hand pose estimator for MANO parameters and camera pose.
- Trained only on HO3D for diffusion fine-tuning, so cross-dataset generalization is evaluated only via DexYCB testing.
Experimental setup
Datasets: HO3D DexYCB
Benchmarks: HO3D DexYCB
Metrics: PSNR SSIM LPIPS Chamfer Distance F-score@5mm F-score@10mm
Baselines: iHOI MonoNHR HandNeRF MOHO Zero123++ Zero123* pixelNeRF HO DDF-HO DreamGaussian
Builds on: Zero123 DreamGaussian ControlNet 3D Gaussian Splatting MANO
Closest papers by meaning
- Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars (NeurIPS 2024 · similarity 0.91)
- HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data (CVPR 2024 · similarity 0.90)
- BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting (CVPR 2025 · similarity 0.89)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.89)
- Learning Explicit Contact for Implicit Reconstruction of Hand-Held Objects from Monocular Images (AAAI 2024 · similarity 0.89)
- HOGSA: Bimanual Hand-Object Interaction Understanding with 3D Gaussian Splatting Based Data Augmentation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.