Research map / Image generation / Editing and style transfer
MARBLE: Material Recomposition and Blending in CLIP-Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MARBLE, which edits object materials in images by injecting CLIP material embeddings into a single UNet block of a frozen SDXL inpainting model, enabling exemplar-based transfer, blending, and parametric control via shallow MLPs trained on rendered data.
Problem
Exemplar-based and fine-grained material editing in single images while preserving geometry and the priors of pre-trained text-to-image diffusion models.
Approach
Injects the CLIP material embedding only into the UNet block responsible for material attribution (near the bottleneck, found by InstantStyle-style probing), keeps the diffusion model frozen, blends materials by interpolating CLIP features, and controls attributes by adding directions predicted by a 2-layer MLP trained on low-rank SVD-approximated CLIP features from synthetic renders.
Key results
- On a 50-object rendered validation set, MARBLE reaches PSNR 26.56 for roughness vs 18.87 for image Concept Slider, and 26.99 vs 19.85 for transparency.
- In a 16-participant user study on 20 real-world images, 87.5% of participants preferred MARBLE over image-based Concept Slider.
- Training the transparency controller with as few as 16 objects gives results comparable to the full 250-object training set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MARBLE instead of Concept Slider (Images) for roughness control | SDXL · Synthetic validation set, roughness | PSNR: 18.87 -> 26.56 | Helped |
| Use MARBLE instead of Concept Slider (Images) for metallic control | SDXL · Synthetic validation set, metallic | PSNR: 19.45 -> 26.82 | Helped |
| Use MARBLE instead of Concept Slider (Images) for transparency control | SDXL · Synthetic validation set, transparency | PSNR: 19.85 -> 26.99 | Helped |
| Use MARBLE instead of Concept Slider (Images) for glow control | SDXL · Synthetic validation set, glow | PSNR: 16.92 -> 19.73 | Helped |
| Inject CLIP features only into the material block rather than all UNet blocks (ablation) | MARBLE material transfer · Material transfer examples | geometry preservation | Helped |
| Train the transparency controller with fewer synthetic objects (ablation) | MARBLE transparency control · Synthetic validation dataset | PSNR | Mixed |
Surprising findings
- Material and style attribution are handled by the same UNet block near the bottleneck, which the authors note is expected since material transfer is a specialized form of style transfer.
- Parametric control transfers to painting styles (SDXL-generated Van Gogh and cyberpunk paintings) despite training only on rendered photorealistic objects.
Limitations
- Parametric control sometimes alters the texture patterns of the object (e.g., leather backpack pattern).
- Increasing transparency or glow can introduce artifacts, especially where no change is expected.
- Quantitative evaluation of parametric control is limited to a synthetic rendered validation set; user study is the only real-image evaluation.
Experimental setup
Datasets: synthetic Blender renders of 300 3D objects with controlled shader properties
Benchmarks: rendered validation set (50 objects) real-world images user study (20 images)
Metrics: PSNR LPIPS CLIP score DreamSim user preference
Baselines: ZeST InstructPix2Pix Concept Slider (Text) Concept Slider (Image)
Builds on: ZeST IP-Adapter InstantStyle Alchemist
Closest papers by meaning
- MatFuse: Controllable Material Generation with Diffusion Models (CVPR 2024 · similarity 0.86)
- Alchemist: Parametric Control of Material Properties with Diffusion Models (CVPR 2024 · similarity 0.85)
- Material Palette: Extraction of Materials from a Single Image (CVPR 2024 · similarity 0.85)
- PhyS-EdiT: Physics-aware Semantic Image Editing with Text Description (CVPR 2025 · similarity 0.85)
- MaRI: Material Retrieval Integration across Domains (CVPR 2025 · similarity 0.85)
- PixPerfect: Seamless Latent Diffusion Local Editing with Discriminative Pixel-Space Refinement (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.