Research map / 3D vision / Depth and stereo
Learned Binocular-Encoding Optics for RGBD Imaging Using Joint Stereo and Focus Cues
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Designs a pair of rank-2 learned diffractive optical elements for a binocular camera and jointly trains them with an IGEV-based stereo matching and U-Net RGBD reconstruction network to recover all-in-focus color and depth over 0.67–8 m.
Problem
Stereo depth estimation and image recovery struggle to capture high-frequency detail and produce high-resolution depth over large depth ranges in realistic cameras, since hardware optics are usually ignored in software design.
Approach
End-to-end co-design of two complementary rank-2 parameterized DOEs placed at the apertures of a stereo pair, with a differentiable image formation model (layered alpha compositing, LS-ASM PSFs) and a stereo network that fuses stereo and focus cues for joint RGB and depth recovery.
Key results
- Rank-2 DOE deep stereo model reaches 32.12 dB PSNR on SceneFlow, 2.96 dB above the thin-lens baseline (29.17 dB at 1.4 m^-1 config).
- EPE reduced to 1.21 px from 1.49 px for the baseline setting, a 0.28 px improvement, per the text.
- Physical prototype with two learned DOEs demonstrates RGBD imaging qualitatively across indoor and outdoor scenes at 1200x1920 resolution, with quantitative results limited to simulation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add rank-2 DOE encoding instead of using no optical encoding | D-S stereo imaging framework · Scene Flow testset, 0.8 m⁻¹ | PSNR: 29.83 -> 32.96 | Helped |
| add rank-2 DOE encoding instead of using no optical encoding | D-S stereo imaging framework · Scene Flow testset, 0.8 m⁻¹ | EPE: 1.38 -> 1.12 | Helped |
| add rank-2 DOE encoding instead of using no optical encoding | D-S stereo imaging framework · Scene Flow testset, 1.4 m⁻¹ | PSNR: 29.17 -> 32.13 | Helped |
| add rank-2 DOE encoding instead of using no optical encoding | D-S stereo imaging framework · Scene Flow testset, 1.4 m⁻¹ | EPE: 1.49 -> 1.21 | Helped |
| replace pixel-wise DOE encoding with rank-2 encoding (ablation) | D-S stereo imaging framework · Scene Flow testset, 0.8 m⁻¹ | PSNR: 31.88 -> 32.96 | Helped |
| replace ring DOE encoding with rank-2 encoding (ablation) | D-S stereo imaging framework · Scene Flow testset, 0.8 m⁻¹ | EPE: 1.16 -> 1.12 | Helped |
| replace rank-1 DOE encoding with rank-2 encoding (ablation) | D-S stereo imaging framework · Scene Flow testset, 0.8 m⁻¹ | PSNR: 32.52 -> 32.96 | Helped |
| replace rank-1 DOE encoding with rank-2 encoding (ablation) | D-S stereo imaging framework · Scene Flow testset, 0.8 m⁻¹ | RMSE: 0.072 -> 0.066 | Helped |
| replace pixel-wise DOE encoding with rank-2 encoding (ablation) | D-S stereo imaging framework · Scene Flow testset, 1.4 m⁻¹ | PSNR: 31.08 -> 32.13 | Helped |
| replace ring DOE encoding with rank-2 encoding (ablation) | D-S stereo imaging framework · Scene Flow testset, 1.4 m⁻¹ | EPE: 1.28 -> 1.21 | Helped |
| replace rank-1 DOE encoding with rank-2 encoding (ablation) | D-S stereo imaging framework · Scene Flow testset, 1.4 m⁻¹ | PSNR: 31.65 -> 32.13 | Helped |
| replace rank-1 DOE encoding with rank-2 encoding (ablation) | D-S stereo imaging framework · Scene Flow testset, 1.4 m⁻¹ | RMSE: 0.079 -> 0.071 | Helped |
Surprising findings
- Symmetric-like network architectures make complementary encoding hard to reach with standard zero or random DOE initialization, so the DOEs are initialized as compound cylindrical lenses.
Limitations
- Computational complexity prevents real-time image reconstruction.
- Artifacts appear in high-dynamic-range scenes such as light sources or specular highlights; rank-1 and rank-2 DOEs produce cross-shaped artifacts from long-tail PSFs due to limited diffraction efficiency.
- Prototype fabrication imperfections cause captured PSFs to deviate from simulation, requiring fine-tuning.
Experimental setup
Datasets: SceneFlow FlyingThings3D InStereo2K
Benchmarks: SceneFlow FlyingThings3D test set InStereo2K test set real-world captures from the stereo prototype
Metrics: PSNR SSIM RMSE EPE (px)
Baselines: IGEV-S Depth from defocus with learned optics (Ikoma et al.) Baseline (thin lens, no DOE) pixel-wise DOE ring DOE rank-1 DOE STTR ACV-Net Tan et al. coded stereo
Builds on: IGEV RAFT-Stereo Ikoma et al. depth-from-defocus with learned optics deep optics LS-ASM
Closest papers by meaning
- Neural RGB(r)D Sensing: Depth and Uncertainty From a Video Camera (CVPR 2019 · similarity 0.88)
- Enhancing the Spatial Resolution of Stereo Images Using a Parallax Prior (CVPR 2018 · similarity 0.86)
- Single-Shot Monocular RGB-D Imaging Using Uneven Double Refraction (CVPR 2020 · similarity 0.86)
- Learning Parallax Attention for Stereo Image Super-Resolution (CVPR 2019 · similarity 0.86)
- Single Image Depth Estimation Trained via Depth From Defocus Cues (CVPR 2019 · similarity 0.85)
- Cross-spectral Gated-RGB Stereo Depth Estimation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.