Research map / 3D vision / Face and shape
Robust 3D Shape Reconstruction in Zero-Shot from a Single Image in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZeroShape-W, a regression model that jointly segments, estimates depth/intrinsics and occluders for occlusion-aware zero-shot single-view 3D shape reconstruction, trained on a generative-model-based synthetic data pipeline with Copy-Paste occluders.
Problem
Zero-shot monocular 3D shape reconstruction methods work on object-segmented, unoccluded images, but degrade on real-world images due to imperfect off-the-shelf segmentation and occlusions.
Approach
A single regression network that jointly predicts visible mask, occluder mask, depth, and camera intrinsics, then reconstructs the full occupancy shape from the visible 3D shape plus occluder silhouette, trained on diverse synthetic images whose foregrounds and backgrounds are produced by ControlNet and object-aware outpainting.
Key results
- On Pix3D, ZeroShape-W achieves FS@τ 38.2 and CD 0.097, beating the strongest baseline ZeroShape w/ SAM + pix2gestalt (33.6 FS, 0.110 CD).
- ZeroShape-W uses 193.7M parameters, less than 1/12 of the parameters of LRM w/ SAM + pix2gestalt (>2400M).
- Category-specific prompts from LLaMA-VID raise Pix3D FS@τ from 38.2 to 39.1 and reduce CD from 0.097 to 0.095.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Unified occlusion-aware regression without off-the-shelf segmentation or amodal completion | ZeroShape-W vs ZeroShape w/ SAM + pix2gestalt · Pix3D | FS@τ: 38.2 vs 33.6 | Helped |
| Unified occlusion-aware regression without off-the-shelf segmentation or amodal completion | ZeroShape-W vs ZeroShape w/ SAM + pix2gestalt · Pix3D | CD: 0.097 vs 0.110 | Helped |
| Category-specific prompt from VLM (LLaMA-VID) instead of category-agnostic prompt | ZeroShape-W · Pix3D | FS@τ: +0.9 (39.1 vs 38.2) | Helped |
| Remove auxiliary depth loss (pseudo depth from Depth Anything V2) (ablation) | ZeroShape-W · Pix3D | FS@τ: 35.9 vs 39.6 full | Hurt |
| Add auxiliary depth loss (ablation) | ZeroShape-W · Pix3D | CD: 0.105 without vs 0.094 full | Helped |
| Add occluder simulation (Copy-Paste augmentation and occluder mask branch) (ablation) | ZeroShape-W · Pix3D | FS@τ: 37.7 without vs 39.6 full | Helped |
| Add text prompts (category-specific priors) (ablation) | ZeroShape-W · Pix3D | FS@τ: 38.0 without vs 39.6 full | Helped |
| Add text prompts (category-specific priors) (ablation) | ZeroShape-W · Pix3D | CD: 0.101 without vs 0.094 full | Helped |
| Unified regression model for occlusion-aware reconstruction vs LRM w/ SAM on occluded images | ZeroShape-W · Pix3D occluded | FS@τ: 34.2 vs 29.1 | Helped |
Limitations
- Limited training resources force lower resolutions, yielding smoother shapes than LRM.
- Struggles with objects clipped by image boundaries.
- Copy-Paste occluders are unrealistic and introduce a domain gap.
Experimental setup
Datasets: ShapeNetCore.v2 Objaverse-LVIS Pix3D ObjectNet OfficeHome PACS Places
Benchmarks: Pix3D
Metrics: F-Score Chamfer Distance
Baselines: LRM w/ SAM ZeroShape w/ SAM LRM w/ SAM + pix2gestalt ZeroShape w/ SAM + pix2gestalt
Builds on: ZeroShape LRM DPT ControlNet Copy-Paste augmentation
Closest papers by meaning
- ZeroShape: Regression-based Zero-shot Shape Reconstruction (CVPR 2024 · similarity 0.90)
- Towards In-the-wild 3D Plane Reconstruction from a Single Image (CVPR 2025 · similarity 0.87)
- LRM-Zero: Training Large Reconstruction Models with Synthesized Data (NeurIPS 2024 · similarity 0.87)
- Slice3D: Multi-Slice Occlusion-Revealing Single View 3D Reconstruction (CVPR 2024 · similarity 0.86)
- WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects Under Occlusion (CVPR 2024 · similarity 0.86)
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.