Research map / 3D vision / Depth and stereo
SfPUEL: Shape from Polarization under Unknown Environment Light
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SfPUEL, a transformer-based end-to-end shape-from-polarization method that jointly predicts surface normals and metallic/dielectric material segmentation under unknown environment light, using photometric-stereo priors from SDM-UniPS and a DoLP-masked cross-attention fusion.
Problem
Single-shot surface normal estimation from polarization images under unknown environment illumination, where specular and diffuse reflections blend and AoLP maps become ambiguous and material-dependent.
Approach
Combines a frozen photometric-stereo pretrained encoder (SDM-UniPS) with a trainable polarization encoder via DoLP-guided masked cross-attention, followed by image-level and pixel-level transformer attention, and jointly predicts material segmentation to aid normal estimation.
Key results
- On synthetic data, SfPUEL achieves 12.39° mean angular error versus 20.33° for DSINE and 20.60° for DeepSfP, with 97.8% material accuracy.
- On the real-world dataset, SfPUEL reaches 11.16° mean angular error and 61.2% accuracy at 11.25°, versus 21.98° for DeepSfP.
- On the PANDORA dataset, SfPUEL reports 16.89° mean angular error, better than DeepSfP (22.77°) and SfPW (41.08°) but worse than DSINE (17.75°) on mean error.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SfPUEL instead of DSINE | Single-shot normal estimation · Synthetic data | mean angular error (°): 20.33 -> 12.39 | Helped |
| Use SfPUEL instead of DSINE | Single-shot normal estimation · Our real-world dataset | mean angular error (°): 21.61 -> 11.16 | Helped |
| Use SfPUEL instead of DSINE | Single-shot normal estimation · PANDORA dataset | mean angular error (°): 17.75 -> 16.89 | Helped |
| Use four polarization images instead of one (ablation) | SDM-UniPS · Our real-world dataset | mean angular error (°): 19.46 -> 15.73 | Helped |
| Add PolFEM rather than finetuning PSPEM (ablation) | SfPUEL · Our real-world dataset | mean angular error (°): 12.11 -> 11.16 | Helped |
| Add the DoLP-guided mask to cross-attention (ablation) | SfPUEL · Our real-world dataset | mean angular error (°): 11.86 -> 11.16 | Helped |
| Add material segmentation estimation (ablation) | SfPUEL · Our real-world dataset | mean angular error (°): 12.02 -> 11.16 | Helped |
| Add the polarization encoder (ablation) | SfPUEL · Our real-world dataset | mean angular error (°): 11.33 -> 11.16 | Helped |
Surprising findings
- Pretrained photometric-stereo model SDM-UniPS with a single polarization image still gives plausible normals, and performance improves with more polarization inputs.
- Training SfPUEL from Xavier initialization failed to converge after over 80 epochs, so pretrained SDM-UniPS weights were needed.
Limitations
- Requires an object mask as input, obtained with SAM in experiments.
- Training uses unpolarized environment maps since large-scale polarized environment light data is unavailable.
- Performance degrades on rough or depolarizing surfaces (e.g., stone, fabric) and over-exposed regions where polarization cues are weak.
Experimental setup
Datasets: synthetic SfP dataset rendered with Mitsuba2 (20,000 training sets) PANDORA dataset
Benchmarks: synthetic SfP dataset (1,000 samples) SfPUEL real-world dataset PANDORA dataset
Metrics: mean angular error median angular error RMSE angular accuracy at 11.25/22.5/30 degrees material classification accuracy
Baselines: SfPW DeepSfP UNE DSINE One-2-3-45 PANDORA
Builds on: SDM-UniPS UPerNet ConvNeXt DeepSfP efficient additive attention
Closest papers by meaning
- Shape From Polarization for Complex Scenes in the Wild (CVPR 2022 · similarity 0.91)
- Glossy Object Reconstruction with Cost-effective Polarized Acquisition (CVPR 2025 · similarity 0.84)
- Material Palette: Extraction of Materials from a Single Image (CVPR 2024 · similarity 0.84)
- SPIDeRS: Structured Polarization for Invisible Depth and Reflectance Sensing (CVPR 2024 · similarity 0.84)
- Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion (CVPR 2025 · similarity 0.84)
- PolarFree: Polarization-based Reflection-Free Imaging (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.