Research map / 3D vision / 3D generation
GenZI: Zero-Shot 3D Human-Scene Interaction Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Zero-shot text-driven synthesis of 3D humans interacting with 3D scenes by distilling 2D interaction priors from Stable Diffusion inpainting into a robust multi-view SMPL-X fitting, with no 3D interaction training data.
Problem
Synthesizing plausible 3D human-scene interactions from text normally requires supervised training on scarce captured 3D human-scene interaction datasets, which limits generalization to diverse scenes and actions.
Approach
Use a latent diffusion inpainting VLM to imagine humans in multiple rendered scene views, with a cross-attention-based dynamic mask instead of manual masks; lift the 2D poses to 3D SMPL-X via robust view-weighted optimization with scene and body priors, and iteratively refine using rendered silhouettes as masks.
Key results
- In the binary perceptual study, participants preferred GenZI over all baselines more than 87% of the time.
- GenZI achieved the best CLIP semantic consistency score of 0.2710, versus 0.2598 for Hassan et al.
- Unary realism rating for GenZI averaged 3.6 out of 5, while all baselines scored below 2.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dynamic masking instead of random fixed masking (ablation) | GenZI · Sketchfab dataset | CLIP: 0.2639 -> 0.2710 | Helped |
| Adaptively select consistent views instead of fixing view weights to w = 1 (ablation) | GenZI · Sketchfab dataset | CLIP: 0.2694 -> 0.2710 | Helped |
| Iteratively refine the 2D inpainting and 3D lifting (ablation) | GenZI · Sketchfab dataset | CLIP: 0.2664 -> 0.2710 | Helped |
Surprising findings
- Single-view baselines achieve higher diversity entropy and cluster size but lower semantic plausibility, suggesting diversity metrics do not reflect perceptual quality.
Limitations
- Limited by the inpainting capability of latent diffusion models to imagine plausible 2D human-scene compositions.
- Diffusion models are slow at inference due to iterative sampling.
- Evaluated on only 8 Sketchfab scenes and 38 actions.
Experimental setup
Datasets: Sketchfab 3D scenes (8 scenes, 38 actions)
Benchmarks: Sketchfab dataset (8 scenes, 38 actions)
Metrics: CLIP score entropy cluster size non-collision contact user study binary preference user study unary realism rating
Baselines: COINS Hassan et al. (PROX-based 3D human estimation) Ours-Single View
Builds on: Stable Diffusion Inpainting SMPLify-X VPoser AlphaPose COINS
Closest papers by meaning
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.90)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.88)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.87)
- HSI-GPT: A General-Purpose Large Scene-Motion-Language Model for Human Scene Interaction (CVPR 2025 · similarity 0.87)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.87)
- SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.