Research map / 3D vision / Face and shape
MeshGen: Generating PBR Textured Mesh with Render-Enhanced Auto-Encoder and Generative Data Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MeshGen, an image-to-3D pipeline that generates shapes via a render-enhanced point-to-shape triplane auto-encoder and augmented diffusion, then produces PBR textures with reference-attention multi-view ControlNet, PBR decomposer, and UV inpainter.
Problem
Existing 3D-native diffusion models produce overly simple, symmetric shapes that poorly match reference images, have weak auto-encoders lacking surface detail, and image-conditioned texturing yields baked-in lighting and inconsistent results rather than PBR materials.
Approach
Adds render-based perceptual loss (normal maps) with ray-based empty-space regularization and coarse-to-fine training to a point-to-triplane auto-encoder; trains image-to-shape diffusion with geometric alignment (rotating point clouds to match views) and generative rendering augmentation (ControlNet + IC-Light); and uses reference-attention multi-view generation, a diffusion PBR decomposer, and a
Key results
- On GSO, MeshGen reaches F-Score 0.971 and CD 0.028, beating MeshFormer (0.963 / 0.031).
- On OmniObject3D, MeshGen reaches F-Score 0.918 and CD 0.040 versus MeshFormer's 0.914 / 0.043.
- In a 52-person user study, MeshGen won 92.31% on image alignment and 82.69% on overall quality against the next-best texturing method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add render-based perceptual refinement to the auto-encoder (ablation) | MeshGen auto-encoder · Objaverse test set | Accuracy: 94.745 -> 96.987 | Helped |
| add render-based perceptual refinement to the auto-encoder (ablation) | MeshGen auto-encoder · Objaverse test set | VolumeIoU: 87.164 -> 91.045 | Helped |
| remove render-based perceptual loss (ablation) | MeshGen auto-encoder · 3D mesh reconstruction | geometric detail quality | Hurt |
| add geometric alignment augmentation (ablation) | MeshGen image-to-shape diffusion · Image-to-shape generation | image-shape alignment | Helped |
| remove geometric alignment augmentation (ablation) | MeshGen image-to-shape diffusion · Image-to-shape generation | image-shape alignment | Hurt |
| add generative rendering augmentation (ablation) | MeshGen image-to-shape diffusion · Image-to-shape generation | lighting understanding | Helped |
| remove generative rendering augmentation (ablation) | MeshGen image-to-shape diffusion · Image-to-shape generation | lighting understanding | Hurt |
| fine-tune reference-attention key and value projections on translated and rotated condition images (ablation) | MeshGen multi-view texture generator · Image-conditioned texture generation | reference-image consistency | Helped |
| fully fine-tune the texture generator instead of only reference-attention projections (ablation) | MeshGen multi-view texture generator · Image-conditioned texture generation | texture detail quality | Hurt |
Limitations
- Texture quality limitations
- Handling of transparent objects
- Comparison to MeshFormer relies on official demo since code is not open-sourced
Experimental setup
Datasets: Objaverse GObjaverse GSO OmniObject3D
Benchmarks: GSO OmniObject3D Objaverse test set
Metrics: F-Score Chamfer distance accuracy volume IoU user win rate
Baselines: InstantMesh TripoSR MeshFormer CraftsMan LN3Diff 3DTopia-XL Paint3D EASI-Tex TEXTure
Builds on: 3DShape2Vecset Rodin Stable Diffusion ControlNet IC-Light
Closest papers by meaning
- Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials (NeurIPS 2024 · similarity 0.91)
- ARMesh: Autoregressive Mesh Generation via Next-Level-of-Detail Prediction (NeurIPS 2025 · similarity 0.90)
- MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers (ICLR 2025 · similarity 0.89)
- TexGaussian: Generating High-quality PBR Material via Octree-based 3D Gaussian Splatting (CVPR 2025 · similarity 0.89)
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.89)
- PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.