Research map / 3D vision / Face and shape
Multistable Shape from Shading Emerges from Patch Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a small pixel-space conditional diffusion model on 16x16 patches of shading images and samples surface normals multi-scale with inter-patch consistency and lighting guidance to produce multistable shape distributions from a single shading image.
Problem
Monocular shape-from-shading models produce point estimates or unimodal distributions, failing to represent the inherent convex/concave and continuous ambiguities that humans perceive as multistable.
Approach
A bottom-up patch-level conditional diffusion model for surface normals, deployed patch-wise with inter-patch curvature/integrability guidance, multiscale fine-coarse-fine V-cycle sampling with noise-and-resume, and a discrete dominant-light-direction guidance step that flips patches to be consistent with a global lighting direction.
Key results
- On multistable perception stimuli, the model's Wasserstein distance to the two-mode reference is 12.59 for four circles vs 30.94 for Wonder3D.
- On the SfS benchmark, the model's average top-5 median angular error is competitive (e.g., 9.99 on turtle vs 9.59 for Wonder3D), roughly on par with best existing methods.
- Model is small (about 10MB weights) and trains in about 40 hours on one A100; inference takes about 105-125 s on a Quadro RTX 8000.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add convex/concave flip data augmentation during training (ablation) | Patch Diffusion · Multistable perception stimuli: four circles, nested rings, star, and snake | Wasserstein distance: 33.79 -> 12.59; 39.56 -> 29.96; 22.04 -> 17.32; 23.02 -> 18.27 | Helped |
| add convex/concave flip data augmentation during training (ablation) | Patch Diffusion · Shape from shading benchmark | median angular error: 11.49 -> 14.95; 15.56 -> 21.46; 14.17 -> 15.94; 9.27 -> 11.98; 8.72 -> 9.99; 22.01 -> 27.21 | Mixed |
| remove lighting consistency guidance (ablation) | Multiscale Patch Diffusion · Ambiguous images with multiple regions | global mode consistency | Hurt |
| add multiscale sampling instead of single-scale spatial guidance (ablation) | Patch Diffusion · Ambiguous shape-from-shading images | global shape consistency | Helped |
| add spatial consistency guidance to independent patch reconstruction (ablation) | Patch Diffusion · Ambiguous shape-from-shading images | inter-patch consistency | Helped |
| add lighting consistency guidance after multiscale sampling (ablation) | Patch Diffusion · Ambiguous images with multiple regions | global convex/concave mode consistency | Helped |
| bias training lighting so 80% of images are lit from above (ablation) | Patch Diffusion Model B · Ambiguous test image | balance of convex and concave predictions | Hurt |
Surprising findings
- Without any convex/concave flip augmentation the model already exhibits multistable outputs on ambiguous test images.
- A model trained on everyday objects does not show the four-mode quadratic-surface ambiguity predicted by theory; the four modes appear only when trained on random cubic splines.
Limitations
- Restricted to textureless, shadowless Lambertian shading; ignores glossy highlights, cast shadows and texture cues.
- Predominantly bottom-up, so it struggles with large cast-shadow regions needing non-local context.
- V-cycle multiscale sampling scales linearly with number of resolutions and requires manual schedule search.
Experimental setup
Datasets: UniPS dataset Adobe3D Assets
Benchmarks: shape from shading dataset (Xiong et al.) multistable perception stimuli captured ambiguous images Wasserstein distance on t-SNE embeddings
Metrics: median angular error 1-Wasserstein distance
Baselines: SIRFS Derender3D Wonder3D Marigold Depth Anything Xiong et al. known-lighting method
Builds on: DDPM DDIM Wonder3D SIRFS
Closest papers by meaning
- Generative Perception of Shape and Material from Differential Motion (NeurIPS 2025 · similarity 0.89)
- Denoising Functional Maps: Diffusion Models for Shape Correspondence (CVPR 2025 · similarity 0.86)
- Adaptive 3D Reconstruction via Diffusion Priors and Forward Curvature-Matching Likelihood Updates (NeurIPS 2025 · similarity 0.86)
- Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models (NeurIPS 2024 · similarity 0.86)
- DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans (CVPR 2024 · similarity 0.86)
- Coherent 3D Scene Diffusion From a Single RGB Image (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.