Research map / 3D vision / Face and shape
Fancy123: One Image to High-Quality 3D Mesh Generation via Plug-and-Play Deformation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fancy123 improves single-image-to-3D mesh generation by adding plug-and-play 2D image deformation (multiview consistency), 3D Jacobian-field mesh deformation (input fidelity), and unprojection of images onto the mesh (clear colors) at inference time.
Problem
Single-image-to-3D mesh generation suffers from multiview inconsistency among diffusion-generated views, low fidelity to the input image, and blurry mesh coloration.
Approach
Optimize grid-based 2D deformation fields on multiview images so they align before unprojection onto an LRM mesh, then optimize a Jacobian-field 3D deformation to match the input image and unproject the input image onto the mesh for clear colors; both modules are training-free, inference-time, and plug-and-play.
Key results
- Fancy123 achieves FID 37.99 vs 46.16 for InstantMesh and 49.89 for SF3D on GSO, best on 6 of 7 metrics.
- Fancy123 reaches CLIP-Similarity 0.835 vs 0.800 for InstantMesh, the best appearance result across all six appearance metrics.
- The full pipeline takes about 62 seconds on an NVIDIA A100 GPU with InstantMesh as backbone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 2D deformation to multiview images before unprojection (ablation) | InstantMesh · GSO | appearance clarity | Helped |
| unproject multiview images without 2D deformation (ablation) | InstantMesh · GSO | appearance clarity | Mixed |
| add 3D mesh deformation before unprojecting the input image (ablation) | InstantMesh · GSO | input-image fidelity | Helped |
| unproject the input image without first deforming the mesh (ablation) | InstantMesh · GSO | input-image fidelity | Hurt |
| use Jacobian-based deformation instead of vertex replacement or a 3D deformation field (ablation) | InstantMesh · GSO | cross-view plausibility | Helped |
| remove Laplacian smooth loss from Jacobian-based deformation (ablation) | InstantMesh · GSO | cross-view plausibility | No clear effect |
| apply fidelity enhancement to TripoSR | TripoSR · single-image-to-3D mesh | visual reconstruction quality | Helped |
| apply appearance enhancement to Unique3D | Unique3D · single-image-to-3D mesh | visual reconstruction quality | Helped |
| apply the complete refinement pipeline to LGM | LGM · single-image-to-3D mesh | visual reconstruction quality | Helped |
| use Fancy123 instead of InstantMesh | InstantMesh · GSO | FID: 46.16 -> 37.99 | Helped |
| use Fancy123 instead of SF3D | SF3D · GSO | LPIPS: 0.343 -> 0.330 | Helped |
| use Fancy123 instead of SF3D | SF3D · GSO | PSNR: 14.32 -> 14.37 | Helped |
Surprising findings
- PSNR/LPIPS-style metrics disagree with human visual judgment of appearance quality, so the authors recommend visual comparison over metrics.
Limitations
- Performance is constrained by the adopted multiview diffusion model.
- 3D deformation is supervised only by RGBA images, causing artifacts when different semantic parts share similar colors.
- Standard image-similarity and geometry metrics do not align well with human perception.
Experimental setup
Datasets: Google Scanned Objects
Benchmarks: Google Scanned Objects
Metrics: FID LPIPS PSNR SSIM CLIP-Similarity Chamfer Distance F-Score
Baselines: LGM TripoSR CRM InstantMesh Unique3D SF3D
Builds on: InstantMesh Unique3D APAP Zero123++
Closest papers by meaning
- Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (ICLR 2024 · similarity 0.87)
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.87)
- Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image (NeurIPS 2024 · similarity 0.87)
- Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle (AAAI 2025 · similarity 0.85)
- Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generation (CVPR 2025 · similarity 0.85)
- MeshFormer : High-Quality Mesh Generation with 3D-Guided Reconstruction Model (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.