Research map / Image generation / Editing and style transfer
PhyS-EdiT: Physics-aware Semantic Image Editing with Text Description
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PhyS-EdiT, a dual-network diffusion editor (initialized from InstructPix2Pix) that jointly controls material (roughness, metallicity, albedo, transparency), lighting, and text-driven semantics, trained on the new synthetic PR-TIPS dataset.
Problem
Joint, disentangled text-guided editing of material properties, lighting, and high-level semantics in images, which prior physics-aware editors handle only separately.
Approach
A dual-denoising-network diffusion model with a low-level U-Net conditioned on material and lighting condition maps (via ControlNet and feature-map multiplication), a high-level text-conditioned U-Net, and a fusion network, trained sequentially (material, then lighting, then semantics) with frozen weights to preserve generative priors.
Key results
- Material editing: PSNR 26.01, LPIPS 0.15, FID 23.38 vs InstructPix2Pix PSNR 20.85, LPIPS 0.31, FID 40.80.
- Lighting editing: PSNR 26.96, LPIPS 0.13, FID 21.59 vs DiLightNet PSNR 17.15, LPIPS 0.39, FID 51.08.
- Semantic editing: CLIP 0.85 vs 0.81 for InstructPix2Pix and Stable Diffusion 3; user study gives 68% material preference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add weight freezing during initial training (ablation) | PhyS-EdiT · Semantic editing | CLIP: 0.76 -> 0.85 | Helped |
| Add weight freezing during initial training (ablation) | PhyS-EdiT · Physical editing | LPIPS: 0.38 -> 0.36 | Helped |
| Jointly train material and lighting editing components (ablation) | PhyS-EdiT · Physical editing | PSNR: 25.80 -> 26.39 | Helped |
| Jointly train material and lighting editing components (ablation) | PhyS-EdiT · Semantic editing | LPIPS: 0.40 -> 0.39 | Helped |
Surprising findings
- Removing weight freezing yields a slightly better physical LPIPS (0.38 vs 0.36 full model are different metrics; w/o WF physical LPIPS 0.38 vs full 0.36 is worse) but lower CLIP, so the model ignores semantic instructions.
Limitations
- Limited to single-object editing; poor generalization to full scene-level editing.
- Requires two forward passes, making it slower than single-pass approaches.
- Real-image lighting conditions estimated with an off-the-shelf estimator, which may be inaccurate.
Experimental setup
Datasets: PR-TIPS InstructPix2Pix editing dataset Objaverse Polyhaven HDRI maps
Benchmarks: PR-TIPS test split user study with 20 participants
Metrics: PSNR SSIM LPIPS FID CLIP score
Baselines: InstructPix2Pix Subias et al. DiLightNet Stable Diffusion 3
Builds on: InstructPix2Pix ControlNet DiLightNet Stable Diffusion
Closest papers by meaning
- Alchemist: Parametric Control of Material Properties with Diffusion Models (CVPR 2024 · similarity 0.88)
- Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.85)
- MARBLE: Material Recomposition and Blending in CLIP-Space (CVPR 2025 · similarity 0.85)
- An Item Is Worth a Prompt: Versatile Image Editing with Disentangled Control (AAAI 2025 · similarity 0.85)
- EditBridge: Towards Faithful and Efficient Ultra-High-Resolution Image Editing (NeurIPS 2026 · similarity 0.84)
- PIXELS: Progressive Image Xemplar-based Editing with Latent Surgery (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.