Research map / 3D vision / Face and shape
Pygmalion Effect in Vision: Image-to-Clay Translation for Reflective Geometry Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts reflective multi-view images into matte clay-style renders with a fine-tuned OminiControl (FLUX) translator and adds a clay-guided branch to Reflective Gaussian Splatting to improve mesh geometry and normals.
Problem
Reconstructing geometry of reflective objects from multi-view images is hard because view-dependent specular reflections entangle appearance with shape, breaking view-invariant radiance consistency.
Approach
Translate reflective training images into neutral white-clay renders with an image-to-image model, and use them as a reflection-free supervision branch that shares Gaussian geometry with a BRDF-based reflective branch in 2DGS-based reconstruction.
Key results
- On Glossy Synthetic, the method reports average Chamfer-L1 0.0061, the lowest among listed methods (RGS 0.0085, GS-2DGS 0.0068).
- On DTU, average Chamfer distance 0.74 versus 0.84 for RGS and 0.80 for 2DGS.
- Image-to-clay translation on 300 Objaverse test images reaches PSNR 24.27 versus 23.87 for Nano-Banana and 21.21 for Qwen-Image-Edit; photometric novel-view quality is described as competitive rather than best.
Surprising findings
- Generating clay images first and then reflective ones produced fewer artifacts than the reverse direction.
- Some scenes (Tbell, Teapot, Bell) fall back to baseline-like behavior once clay supervision is removed, so they are trained only with the clay stage.
Limitations
- Incorrect image-to-clay translations (e.g., coffee inside a mug) propagate errors into geometry learning.
- Gains are smaller on objects whose geometry is already easy to infer from BRDF estimation; MAE on Shiny Blender is higher than some baselines.
- Relies on a translation model; dependence on training distribution of materials and lighting.
Experimental setup
Datasets: Objaverse PolyHaven Shiny Blender Glossy Synthetic Ref-Real DTU
Benchmarks: Glossy Synthetic DTU Shiny Blender Ref-Real
Metrics: Chamfer distance mean angular error PSNR SSIM LPIPS PSNR/SSIM/LPIPS for clay translation
Baselines: TensoSDF NeRO GShader GS-IR R3DG Ref-GS RGS GS-2DGS 3DGS-DR 2DGS 3DGS SuGaR
Builds on: Reflective Gaussian Splatting 2D Gaussian Splatting OminiControl FLUX LoRA
Closest papers by meaning
- MaterialRefGS: Reflective Gaussian Splatting with Multi-view Consistent Material Inference (NeurIPS 2025 · similarity 0.87)
- UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections (NeurIPS 2024 · similarity 0.87)
- GNeRP: Gaussian-guided Neural Reconstruction of Reflective Objects with Noisy Polarization Priors (ICLR 2024 · similarity 0.87)
- Differentiable Neural Surface Refinement for Modeling Transparent Objects (CVPR 2024 · similarity 0.86)
- Reflective Gaussian Splatting (ICLR 2025 · similarity 0.86)
- Geometry Field Splatting with Gaussian Surfels (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.