Research map / Vision-language and multimodal / Multimodal LLMs
Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Make-it-Real uses GPT-4V to recognize and assign real-world PBR materials from a 1,400-material library to segmented parts of albedo-only 3D meshes, then generates full SVBRDF maps via albedo-referenced nearest-neighbor mapping.
Problem
Albedo-only 3D assets and generated 3D objects lack physically realistic materials (roughness, metallic, normal, height, displacement), and manual material assignment is slow and requires specialized software.
Approach
Use GPT-4V's visual recognition and prior knowledge, guided by Set-of-Mark visual cues and hierarchical text prompts over a captioned material library, to match real materials to mesh segments, then transfer SVBRDF maps to the albedo via a KD-tree pixel-level albedo-referenced estimation.
Key results
- Make-it-Real refined Objaverse assets are preferred 84.8% by GPT-4V evaluation versus 15.2% for base objects.
- User preference for refined TripoSR outputs is 77.0% versus 33.0% for base objects.
- Refinement is preferred in all evaluated categories, with the smallest margin for Fantasia3D (53.8% GPT, 51.3% user).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Refine Objaverse objects with Make-it-Real | Objaverse assets · Objaverse | GPT evaluation preference: 15.2% -> 84.8% | Helped |
| Refine Objaverse objects with Make-it-Real | Objaverse assets · Objaverse | User preference: 22.2% -> 77.8% | Helped |
| Refine InstantMesh objects with Make-it-Real | InstantMesh · InstantMesh | GPT evaluation preference: 28.2% -> 71.8% | Helped |
| Refine InstantMesh objects with Make-it-Real | InstantMesh · InstantMesh | User preference: 31.1% -> 68.9% | Helped |
| Refine TripoSR objects with Make-it-Real | TripoSR · TripoSR | GPT evaluation preference: 36.4% -> 63.6% | Helped |
| Refine TripoSR objects with Make-it-Real | TripoSR · TripoSR | User preference: 33.0% -> 77.0% | Helped |
| Refine Instant3D objects with Make-it-Real | Instant3D · Instant3D | GPT evaluation preference: 38.5% -> 61.5% | Helped |
| Refine Instant3D objects with Make-it-Real | Instant3D · Instant3D | User preference: 35.4% -> 64.6% | Helped |
| Refine MVDream objects with Make-it-Real | MVDream · MVDream | GPT evaluation preference: 44.1% -> 55.9% | Helped |
| Refine MVDream objects with Make-it-Real | MVDream · MVDream | User preference: 41.5% -> 58.5% | Helped |
| Refine Fantasia3D objects with Make-it-Real | Fantasia3D · Fantasia3D | GPT evaluation preference: 46.2% -> 53.8% | Helped |
| Refine Fantasia3D objects with Make-it-Real | Fantasia3D · Fantasia3D | User preference: 48.7% -> 51.3% | Helped |
Surprising findings
- The paper reports that the GPT-4V evaluation agrees with human preference across all evaluated 3D generation methods.
Limitations
- Cannot convert shaded texture maps back to albedo, so shadows and highlights can be misassigned materials for already-lit meshes.
- Low-quality base meshes make MLLM material identification difficult without ground-truth text descriptions.
- Evaluation relies on GPT-4V pairwise judgments and a user study rather than physically grounded metrics.
Experimental setup
Datasets: Objaverse CC0 PBR material library (1,400 materials, 13 categories, 80 subtypes)
Benchmarks: Objaverse 200-object subset InstantMesh/TripoSR/MVDream/Instant3D/Fantasia3D generated objects (200 each)
Metrics: GPT-4V pairwise preference user preference rate
Baselines: Material Palette Paint-it Matlaber TANGO Fantasia3D
Builds on: Semantic-SAM Set-of-Mark Kaolin Material Palette
Closest papers by meaning
- VLMaterial: Procedural Material Generation with Large Vision-Language Models (ICLR 2025 · similarity 0.87)
- Material Palette: Extraction of Materials from a Single Image (CVPR 2024 · similarity 0.87)
- Hi3DEval: Advancing 3D Generation Evaluation with Hierarchical Validity (NeurIPS 2025 · similarity 0.87)
- Material Anything: Generating Materials for Any 3D Object via Diffusion (CVPR 2025 · similarity 0.86)
- Structured 3D Latents for Scalable and Versatile 3D Generation (CVPR 2025 · similarity 0.86)
- MatSynth: A Modern PBR Materials Dataset (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.