Research map / 3D vision / Face and shape
Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Meta 3D AssetGen generates four-view shaded+albedo grids with a text-to-image model, then reconstructs SDF geometry and PBR materials (albedo, metalness, roughness) with a deferred-shading-supervised LRM and a UV-space texture refiner.
Problem
Text-to-3D generators are slow and produce blurry meshes that bake lighting into albedo, so generated assets cannot be relit; existing feed-forward few-view reconstructors do not output PBR materials.
Approach
Generate a 4-view 6-channel grid (shaded + albedo) so the deterministic reconstructor can infer PBR materials; predict an SDF instead of opacity, supervise PBR with a cheap deferred shading loss, and refine the UV texture with a cross-view attention transformer/U-Net.
Key results
- Meta-ILRM achieves LPIPS 0.057 and CD 1.137 (x1e-2) on GSO four-view reconstruction vs. 0.100 / 1.364 for best baselines (GRM, InstantMesh).
- Texture refinement and albedo+shaded input raise PBR albedo PSNR from 21.12 to 24.39 and metalness PSNR from 18.64 to 27.28 in ablations.
- In a crowdsourced text-to-3D study (11,080 responses), AssetGen wins 72% vs. the best comparable-speed industry competitors, and 94.6% visual quality vs. Meshy v3 preview.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the base density representation with VolSDF rendering (ablation) | LightplaneLRM · GSO four-view reconstruction | Depth: 0.456 -> 0.201 | Helped |
| Add direct SDF loss (ablation) | LightplaneLRM + VolSDF rendering · GSO four-view reconstruction | Chamfer Distance: 1.212 -> 1.137 | Helped |
| Add texture refinement (ablation) | LightplaneLRM + VolSDF rendering + Direct SDF loss · GSO four-view reconstruction | LPIPS: 0.083 -> 0.057 | Helped |
| Add material prediction (ablation) | LightplaneLRM w/ SDF · Four-view PBR reconstruction | LPIPS albedo: 0.117 -> 0.097 | Helped |
| Add deferred shading loss (ablation) | Material prediction model · Four-view PBR reconstruction | PSNR metal: 15.99 -> 18.64 | Helped |
| Add texture refinement (ablation) | Material prediction + Deferred shading loss · Four-view PBR reconstruction | PSNR metal: 18.64 -> 22.19 | Helped |
| Add albedo and shaded input channels (ablation) | MetaILRM · Four-view PBR reconstruction | PSNR albedo: 21.12 -> 23.02 | Helped |
| Add texture refinement after albedo and shaded input (ablation) | MetaILRM with albedo and shaded input · Four-view PBR reconstruction | PSNR metal: 20.43 -> 27.28 | Helped |
| Use SDF-based reconstruction with direct SDF supervision rather than the LightplaneLRM base (ablation) | MetaILRM · Four-view reconstruction on GSO | Chamfer Distance: 1.313 -> 1.137 | Helped |
| Use AssetGen rather than GRM | Text-to-3D generation · Text-to-3D user study | Visual quality win rate: 96.7 % | Helped |
| Use AssetGen rather than InstantMesh | Text-to-3D generation · Text-to-3D user study | Text fidelity win rate: 97.3 % | Helped |
| Use AssetGen rather than Meshy v3 | Text-to-3D generation · Text-to-3D user study | Visual quality win rate: 94.6 % | Helped |
Surprising findings
- Generating the image grid directly with PBR channels (metalness/roughness) was ineffective; using shaded+albedo channels worked better.
- Against slower refined baselines, AssetGen was on par with Luma Genie in text fidelity while being about 20x faster.
Limitations
- Multi-view grid is not guaranteed to be view-consistent, which can hurt reconstruction.
- SDF representation may mis-model translucent objects and thin structures like hair or fur.
- Only albedo, metalness and roughness are predicted; no emissivity or ambient occlusion; tested only on object-level assets.
Experimental setup
Datasets: internal 140K artist-created mesh dataset Google Scanned Objects (332 meshes subset) internal 256-mesh PBR artist dataset DreamFusion prompt set (404 prompts)
Benchmarks: Google Scanned Objects (4-view reconstruction) internal PBR artist dataset DreamFusion 404-prompt user study
Metrics: LPIPS PSNR depth L1 IoU Chamfer Distance Normal Correctness user win-rate
Baselines: Instant3D-LRM GRM InstantMesh LightplaneLRM Meshy v3 Luma Genie 1.0 MeshLRM
Builds on: Instant3D LightplaneLRM VolSDF Emu
Closest papers by meaning
- MeshGen: Generating PBR Textured Mesh with Render-Enhanced Auto-Encoder and Generative Data Augmentation (CVPR 2025 · similarity 0.91)
- Structured 3D Latents for Scalable and Versatile 3D Generation (CVPR 2025 · similarity 0.88)
- 3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion (CVPR 2025 · similarity 0.87)
- PartGen: Part-level 3D Generation and Reconstruction with Multi-view Diffusion Models (CVPR 2025 · similarity 0.86)
- ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models (CVPR 2024 · similarity 0.86)
- MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.