Research map / Vision-language and multimodal / Multimodal LLMs
VLMaterial: Procedural Material Generation with Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes LLaVA-NeXT (LLaMA 3 8B) to generate Blender procedural material Python programs from a single image, using LLM-based program-level augmentation and MCMC parameter post-optimization, and releases an open 550K-material dataset.
Problem
Inverse procedural material modeling: generating editable procedural material node graphs from a single input image requires expert effort, and prior methods are tied to proprietary software with unreleasable data; zero-shot VLMs fail on this domain-specific task.
Approach
Represent Blender procedural materials as Python programs and fine-tune a pre-trained VLM to generate them from images, expanding a 1.6K artist-made dataset to 550K via LLM-driven evolutionary crossover of programs (FunSearch-style) plus parameter perturbation, followed by gradient-free MCMC parameter refinement.
Key results
- On the Blender test set, the method reaches style loss 0.019, SWD 1.760, CLIP 0.856 and program correctness 0.911, beating GPT-4o-mini (correctness 0.294) and BlenderAlchemy and Nearest Neighbor baselines.
- MCMC post-optimization lowers style loss on real images from 0.025 to 0.018 and on Blender from 0.019 to 0.015.
- In a user study, participants preferred the method over BlenderAlchemy in visual quality 91% and node graph quality 94% of the time on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add graph structure augmentation (ablation) | VLMaterial · Blender | Style loss: 0.032 -> 0.019 | Helped |
| add graph structure augmentation (ablation) | VLMaterial · Substance | Style loss: 0.033 -> 0.026 | Helped |
| add graph structure augmentation (ablation) | VLMaterial · Real images | Style loss: 0.030 -> 0.025 | Helped |
| add graph parameter augmentation (ablation) | VLMaterial · Blender | Style loss: 0.022 -> 0.019 | Helped |
| add graph parameter augmentation (ablation) | VLMaterial · Substance | Style loss: 0.030 -> 0.026 | Helped |
| add graph parameter augmentation (ablation) | VLMaterial · Real images | Style loss: 0.028 -> 0.025 | Helped |
| add MCMC post-optimization (ablation) | VLMaterial · Blender | Style loss: 0.019 -> 0.015 | Helped |
| add MCMC post-optimization (ablation) | VLMaterial · Substance | Style loss: 0.026 -> 0.020 | Helped |
| add MCMC post-optimization (ablation) | VLMaterial · Real images | Style loss: 0.025 -> 0.018 | Helped |
| fine-tune LLaVA-NeXT on the augmented material dataset instead of zero-shot prompting | LLaVA-NeXT · Blender | Style loss: 0.041 -> 0.019 | Helped |
| fine-tune LLaVA-NeXT on the augmented material dataset instead of zero-shot prompting | LLaVA-NeXT · Substance | Style loss: 0.039 -> 0.026 | Helped |
| fine-tune LLaVA-NeXT on the augmented material dataset instead of zero-shot prompting | LLaVA-NeXT · Real images | Style loss: 0.035 -> 0.025 | Hurt |
Surprising findings
- Zero-shot GPT-4o-mini produces valid Blender programs only about 29% of the time on Blender test images, and about 25% on real images.
- Structure augmentation has a bigger effect on program correctness (0.603 to 0.911) than parameter augmentation alone.
Limitations
- Intricate textures exceed the expressiveness of Blender node types within the 30-node limit, causing failures.
- Inference cost is high relative to direct generation because of sampling and post-optimization.
- Performance declines for very complex materials (over 22 nodes or 2,000 tokens) and simple ones with fewer than 4 nodes.
Experimental setup
Datasets: BlenderKit materials Infinigen base materials Blender material packs
Benchmarks: Blender synthetic test set (44 materials) Substance synthetic test set (64 materials) Real photographs (64 images)
Metrics: style loss sliced Wasserstein distance CLIP similarity program correctness
Baselines: GPT-4o-mini Conditional MatFormer BlenderAlchemy Nearest Neighbor retrieval
Builds on: FunSearch LLaVA-NeXT MatFormer BlenderAlchemy LoRA
Closest papers by meaning
- Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials (NeurIPS 2024 · similarity 0.87)
- 3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code (NeurIPS 2026 · similarity 0.87)
- ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling (NeurIPS 2025 · similarity 0.86)
- MatFuse: Controllable Material Generation with Diffusion Models (CVPR 2024 · similarity 0.86)
- SceneCraft: An LLM Agent for Synthesizing 3D Scenes as Blender Code (ICML 2024 · similarity 0.85)
- Material Anything: Generating Materials for Any 3D Object via Diffusion (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.