Research map / 3D vision / Depth and stereo
Differentiable Display Photometric Stereo
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns display illumination patterns end-to-end for display photometric stereo by coupling basis-illumination image formation with an analytic photometric-stereo solver, trained on 3D-printed objects with polarization-based diffuse separation.
Problem
Display photometric stereo relies on heuristic illumination patterns that are sub-optimal for normal reconstruction; the design of display patterns for a specific display-camera system is hard.
Approach
Learn display RGB patterns end-to-end through a differentiable framework that combines weighted-sum basis-image formation with a training-free analytic trinocular photometric-stereo solver, supervised by normals from 3D-printed objects with known geometry, and using LCD polarization with a polarization camera to separate diffuse reflection.
Key results
- Learned patterns reduce average reconstruction error from 0.0805 (best heuristic, group OLAT) to 0.0475 for the group OLAT initialization on the 3D-printed test set.
- Two learned trichromatic patterns (error 0.0476) outperform any tested heuristic design with four patterns.
- Replacing the analytic reconstructor with UniPS raises error from 0.0475 to 0.0951.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn display patterns end-to-end from heuristic or random initial patterns | DDPS · 3D-printed test dataset | reconstruction error: 0.0805 -> 0.0475; 0.0913 -> 0.0443; 0.1461 -> 0.0476 | Helped |
| Initialize learned patterns with flat gray instead of heuristic patterns (ablation) | DDPS · 3D-printed test dataset | reconstruction error: 0.3930 -> 0.0466 | Helped |
| Use 2 learned patterns rather than 4 heuristic patterns (ablation) | DDPS · 3D-printed test dataset | reconstruction error: 0.0476; 0.0805 | Helped |
| Vary the number of learned display patterns from 2 to 5 (ablation) | DDPS · 3D-printed test dataset | reconstruction error: 0.0476; 0.0467; 0.0463; 0.0467 | No clear effect |
| Use inaccurate manually assigned superpixel locations instead of calibrated locations (ablation) | DDPS · 3D-printed test dataset | reconstruction error: 0.0456; 0.0453 | No clear effect |
| Omit intensity fall-off from the image formation model (ablation) | DDPS · 3D-printed test dataset | reconstruction error: 0.0429; 0.0453 | No clear effect |
| Test at varying object depths rather than the assumed fixed distance (ablation) | DDPS · Objects at 40/50/80/100 cm | reconstruction error: 0.0494/0.0417/0.0428/0.0561 | Mixed |
| Separate diffuse and specular reflections using polarization (ablation) | DDPS · Real-world objects | normal reconstruction accuracy | Helped |
| Replace analytic photometric stereo with UniPS (ablation) | DDPS · 3D-printed test dataset | reconstruction error: 0.0475 to 0.0951 | Hurt |
Surprising findings
- Learned patterns converge to similar reconstruction error regardless of initialization, including flat gray noise (0.0466) and random patterns.
- A learning-based photometric stereo (UniPS) substituted into the pipeline degrades accuracy, attributed to poor gradient flow through the network.
Limitations
- Estimates only surface normals; depth reconstruction left for future work.
- Image formation ignores shadows and global illumination; high-speed display-camera synchronization is a challenge.
- Assumes planar scene geometry at fixed 50 cm distance in image formation.
Experimental setup
Datasets: 3D-printed object dataset (40 training scenes, 4 test scenes)
Benchmarks: 3D-printed test dataset (4 scenes)
Metrics: normalized cosine distance reconstruction error
Baselines: OLAT Group OLAT Mono-gradient Mono-complementary Tri-gradient Tri-complementary UniPS SDM-UniPS Bae et al.
Builds on: photometric stereo basis-illumination image formation UniPS
Closest papers by meaning
- Differentiable Adaptive 4D Structured Illumination for Joint Capture of Shape and Reflectance (CVPR 2026 · similarity 0.86)
- EventPS: Real-Time Photometric Stereo Using an Event Camera (CVPR 2024 · similarity 0.85)
- Sparse Views Near Light: A Practical Paradigm for Uncalibrated Point-light Photometric Stereo (CVPR 2024 · similarity 0.85)
- MVCPS-NeuS: Multi-view Constrained Photometric Stereo for Neural Surface Reconstruction (CVPR 2024 · similarity 0.84)
- DiLiGenRT: A Photometric Stereo Dataset with Quantified Roughness and Translucency (CVPR 2024 · similarity 0.84)
- RNb-NeuS: Reflectance and Normal-based Multi-View 3D Reconstruction (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.