Research map / Image generation / Editing and style transfer
MatFuse: Controllable Material Generation with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MatFuse generates and edits SVBRDF material maps with a latent diffusion model conditioned compositionally on palettes, sketches, text and images, using a multi-encoder VQ-GAN with a rendering loss.
Problem
Controllable generation and editing of high-quality 3D materials (SVBRDF maps) is time-consuming and expert-driven, and prior GAN-based methods offer limited control.
Approach
Combines a latent diffusion model with a multi-encoder VQ-GAN that learns a disentangled latent per SVBRDF map, supports multi-condition compositional guidance (global via cross-attention, local via concatenation), and enables map-level "volumetric inpainting" by masking latent channels.
Key results
- MatFuse achieves CLIP-IQA 0.431 and FID 158.53, versus TileGen 0.433 / 184.81 and vanilla LDM 0.269 / 231.64.
- In a pairwise user study with 100 participants, MatFuse received 1078 votes vs. 949 for TileGen (chi-square 16.41, p<0.05).
- The multi-encoder compression model reduces map RMSE substantially versus a single-encoder baseline (e.g., diffuse 0.016 vs 0.047 at codebook 16384; render 0.041 vs 0.227).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LDM with MatFuse | Unconditional material generation · Unconditional material generation | CLIP-IQA: 0.269 ± 0.118 -> 0.431 ± 0.151 | Helped |
| Replace LDM with MatFuse | Unconditional material generation · Unconditional material generation | FID: 231.64 -> 158.53 | Helped |
| Use MatFuse instead of TileGen | Unconditional material generation · Unconditional material generation | CLIP-IQA: 0.433 ± 0.161 -> 0.431 ± 0.151 | No clear effect |
| Use MatFuse instead of TileGen | Unconditional material generation · Unconditional material generation | FID: 184.81 -> 158.53 | Helped |
| Prefer MatFuse over TileGen | Material preference user study · Material preference user study | votes: TileGen=949, MatFuse=1078 | Helped |
| Replace a single encoder with separate map-specific encoders (ablation) | VQ-GAN compression model · Material map reconstruction | RMSE (diffuse): 0.047 -> 0.016 | Helped |
| Replace a single encoder with separate map-specific encoders (ablation) | VQ-GAN compression model · Material map reconstruction | RMSE (normal): 0.051 -> 0.024 | Helped |
| Replace a single encoder with separate map-specific encoders (ablation) | VQ-GAN compression model · Material map reconstruction | RMSE (roughness): 0.102 -> 0.022 | Helped |
| Replace a single encoder with separate map-specific encoders (ablation) | VQ-GAN compression model · Material map reconstruction | RMSE (specular): 0.152 -> 0.020 | Helped |
| Add rendering loss (ablation) | VQ-GAN compression model · Material map reconstruction | RMSE (diffuse): 0.038 -> 0.016 | Helped |
| Add rendering loss (ablation) | VQ-GAN compression model · Material map reconstruction | RMSE (normal): 0.030 -> 0.024 | Helped |
| Add rendering loss (ablation) | VQ-GAN compression model · Material map reconstruction | RMSE (roughness): 0.047 -> 0.022 | Helped |
Surprising findings
- MatFuse is only marginally behind or on par with TileGen on CLIP-IQA despite TileGen being trained on a narrower set of classes, and TileGen's FID is notably worse, which the authors attribute to limited diversity.
Limitations
- High GPU memory cost limits scaling to higher resolutions and fine detail.
- Generated materials are not tileable.
- Evaluation of quality relies on CLIP-IQA and FID, which are not designed for material data; user study limited to comparison with TileGen.
Experimental setup
Datasets: SVBRDF dataset (Deschaintre et al.) PolyHaven materials
Benchmarks: CLIP-IQA on 2,000 unconditional renders FID
Metrics: CLIP-IQA FID RMSE user preference votes chi-square test
Baselines: TileGen LDM (vanilla) MaterialGAN Text2Mat
Builds on: Latent Diffusion Models VQ-GAN Composer classifier-free guidance DDIM
Closest papers by meaning
- Material Anything: Generating Materials for Any 3D Object via Diffusion (CVPR 2025 · similarity 0.88)
- Single Mesh Diffusion Models with Field Latents for Texture Generation (CVPR 2024 · similarity 0.88)
- Material Palette: Extraction of Materials from a Single Image (CVPR 2024 · similarity 0.87)
- Alchemist: Parametric Control of Material Properties with Diffusion Models (CVPR 2024 · similarity 0.86)
- VLMaterial: Procedural Material Generation with Large Vision-Language Models (ICLR 2025 · similarity 0.86)
- Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.