Research map / 3D vision / Face and shape
GenVDM: Generating Vector Displacement Maps From a Single Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates Vector Displacement Maps from a single RGB image by fine-tuning Zero123++ to produce multi-view normal maps, then reconstructing the VDM via neural SDF fitting and a neural deformation field, trained on a 1,200-part dataset extracted from Objaverse.
Problem
Generating vector displacement maps (VDMs), parameterized 3D geometric stamps used in artist workflows, from a single image; existing image-to-3D models generate complete shapes without a parametric 2D domain and depth-based scalar displacement maps cannot capture overhangs or cavities.
Approach
Fine-tune a multi-view diffusion model (Zero123++) on rendered VDM-like patches to output six normal maps, then reconstruct the VDM by fitting a neural SDF to the normals and parameterizing the mesh with an MLP deformation field applied to a square.
Key results
- Ours achieves CLIPImg 0.8520 and 3D-FID 192.7, outperforming Wonder3D (0.8246, 199.5), Magic123, LRM, and scalar DM.
- Ablation: the neural deformation field parameterization gets 3D-FID 192.7 versus 209.9 for topology fixing with Tutte embedding and 217.2 for mesh optimization.
- Full reconstruction pipeline takes about 6 minutes per shape on one A100 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate VDMs with GenVDM rather than the strongest listed baseline | Single-image-to-VDM generation · VDM evaluation set (50 images) | CLIPImg: 0.8246 -> 0.8520 | Helped |
| generate VDMs with GenVDM rather than the strongest listed baseline | Single-image-to-VDM generation · VDM evaluation set (50 images) | CLIPText: 0.2564 -> 0.2701 | Helped |
| generate VDMs with GenVDM rather than the strongest listed baseline | Single-image-to-VDM generation · VDM evaluation set (50 images) | 3D-FID: 199.5 -> 192.7 | Helped |
| use neural deformation instead of topology fixing and Tutte embedding (ablation) | GenVDM VDM reconstruction · VDM reconstruction | CLIPImg: 0.8401 -> 0.8521 | Helped |
| use neural deformation instead of topology fixing and Tutte embedding (ablation) | GenVDM VDM reconstruction · VDM reconstruction | CLIPText: 0.2617 -> 0.2701 | Helped |
| use neural deformation instead of topology fixing and Tutte embedding (ablation) | GenVDM VDM reconstruction · VDM reconstruction | 3D-FID: 209.9 -> 192.7 | Helped |
| use neural deformation instead of fitting a square mesh (ablation) | GenVDM VDM reconstruction · VDM reconstruction | CLIPImg: 0.8245 -> 0.8521 | Helped |
| use neural deformation instead of fitting a square mesh (ablation) | GenVDM VDM reconstruction · VDM reconstruction | CLIPText: 0.2525 -> 0.2701 | Helped |
| use neural deformation instead of fitting a square mesh (ablation) | GenVDM VDM reconstruction · VDM reconstruction | 3D-FID: 217.2 -> 192.7 | Helped |
Surprising findings
- Mesh optimization of a square mesh to the target via differentiable rendering or Chamfer loss falls into local minima and yields large distortion, motivating the neural deformation field.
Limitations
- Per-shape optimization makes inference much slower than feed-forward image-to-3D methods.
- Thin structures are not reconstructed plausibly due to multi-view inconsistency.
- Training data is small (1,200 patches) and focused on organic parts.
Experimental setup
Datasets: Objaverse Objaverse captions
Benchmarks: 50-image VDM evaluation set collected from the Internet and a text-to-image model
Metrics: CLIPImg CLIPText 3D-FID
Baselines: Wonder3D Magic123 LRM Scalar DM (DepthAnything)
Builds on: Zero123++ Wonder3D AtlasNet Screened Poisson Surface Reconstruction
Closest papers by meaning
- ViewCraft3D: High-fidelity and View-Consistent 3D Vector Graphics Synthesis (NeurIPS 2025 · similarity 0.86)
- Enhancing Implicit Shape Generators Using Topological Regularizations (ICML 2024 · similarity 0.85)
- GALA: Geometry-Aware Local Adaptive Grids for Detailed 3D Generation (ICLR 2025 · similarity 0.85)
- AutoPartGen: Autoregressive 3D Part Generation and Discovery (NeurIPS 2025 · similarity 0.84)
- MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers (ICLR 2025 · similarity 0.84)
- MeshArt: Generating Articulated Meshes with Structure-Guided Transformers (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.