Research map / Image generation / Text-to-image diffusion
Alchemist: Parametric Control of Material Properties with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Alchemist fine-tunes InstructPix2Pix (Stable Diffusion 1.5) on a synthetic Blender-rendered material-edit dataset, conditioned on a scalar relative attribute strength, to edit roughness, metallic, albedo, and transparency in real images.
Problem
Fine-grained, continuous control of low-level material properties (roughness, metallic, albedo, transparency) in real images is hard because real datasets with labeled material parameters are scarce and text prompts do not capture continuous material parameters.
Approach
Condition an image-to-image latent diffusion model (initialized from InstructPix2Pix) on the context image, a generic instruction, and a concatenated constant scalar grid giving relative attribute strength, trained on a synthetic dataset of physically-based material edits rendered with Blender so that the scalar gives smooth, parametric control that transfers to real images.
Key results
- On the held-out synthetic set, the method achieves LPIPS of 0.09 for roughness vs 0.13 for prompt-only InstructPix2Pix trained on the same data, with PSNR 31.5 vs 30.9.
- In a user study with N=14 users, the method was preferred overall 70.2% vs 29.8% against the compared baseline.
- The method preferred for photorealism at 69.6% vs 30.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use scalar edit-strength conditioning instead of prompt-only conditioning | Alchemist vs. InstructPix2Pix fine-tuned on the synthetic data · Held-out synthetic dataset, Roughness | LPIPS: 0.13 -> 0.09 | Helped |
| Use scalar edit-strength conditioning instead of prompt-only conditioning | Alchemist vs. InstructPix2Pix fine-tuned on the synthetic data · Held-out synthetic dataset, Metallic | LPIPS: 0.10 -> 0.09 | Helped |
| Use scalar edit-strength conditioning instead of prompt-only conditioning | Alchemist vs. InstructPix2Pix fine-tuned on the synthetic data · Held-out synthetic dataset, Albedo | LPIPS: 0.14 -> 0.10 | Helped |
| Use scalar edit-strength conditioning instead of prompt-only conditioning | Alchemist vs. InstructPix2Pix fine-tuned on the synthetic data · Held-out synthetic dataset, Transparency | LPIPS: 0.13 -> 0.13 | No clear effect |
| Train with joint volume sampling of albedo, roughness, and metallic strengths rather than axis-only sampling (ablation) | Alchemist multi-attribute editing · Joint albedo, roughness, and metallic editing | Qualitative attribute composition | Helped |
| Use Alchemist rather than InstructPix2Pix trained on the synthetic data | Real-image material editing user study · Randomly selected image pairs | Photorealistic edit preference: 69.6% vs. 30.4% | Helped |
| Use Alchemist rather than InstructPix2Pix trained on the synthetic data | Real-image material editing user study · Randomly selected image pairs | Overall edit preference: 70.2% vs. 29.8% | Helped |
Surprising findings
- Prompt-only InstructPix2Pix fine-tuned on the same synthetic data exaggerates edits and yields non-smooth transitions, showing that text-only strength specification is insufficient.
- Training on a volume-sampled multi-attribute dataset was necessary for composing attributes; axis-only sampling biased edits toward a single attribute.
Limitations
- Produces minimal perceptual change for roughness and metallic attributes.
- Occasionally yields physically unrealistic transparency and fails to inpaint consistently (e.g., candy-cane stem omitted).
- Outputs are plausible interpretations rather than exact renderer outcomes.
Experimental setup
Datasets: Synthetic material-edit dataset rendered with Blender Cycles from polyhaven meshes and ambientcg materials DTU MVS (NeRF experiments)
Benchmarks: Held-out synthetic rendered dataset (10 scenes, 15 cameras) Real-image qualitative comparisons DTU NeRF editing
Metrics: PSNR SSIM LPIPS user preference study
Baselines: Subias et al. GAN-based material editing Prompt-to-Prompt with Null-text inversion InstructPix2Pix InstructPix2Pix fine-tuned on synthetic data (prompt-only)
Builds on: InstructPix2Pix Stable Diffusion Prompt-to-Prompt Null-text inversion ControlNet
Closest papers by meaning
- Intrinsic Image Diffusion for Indoor Single-view Material Estimation (CVPR 2024 · similarity 0.89)
- Material Anything: Generating Materials for Any 3D Object via Diffusion (CVPR 2025 · similarity 0.89)
- PhyS-EdiT: Physics-aware Semantic Image Editing with Text Description (CVPR 2025 · similarity 0.88)
- Material Palette: Extraction of Materials from a Single Image (CVPR 2024 · similarity 0.88)
- OMG: Opacity Matters in Material Modeling with Gaussian Splatting (ICLR 2025 · similarity 0.86)
- Eclipse: Disambiguating Illumination and Materials using Unintended Shadows (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.