Research map / 3D vision / Depth and stereo
Robust Depth Enhancement via Polarization Prompt Fusion Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a polarization-guided depth enhancement network that fuses polarization cues into a CompletionFormer backbone pre-trained on RGB-D data via Polarization Prompt Fusion Tuning (PPFT) and a Polarization Prompt Fusion Block.
Problem
Depth sensors (stereo, d-ToF, i-ToF) produce inaccurate depth on transparent, reflective, and textureless regions, and polarization-based depth enhancement has been limited to physics-based methods for single sensors, with small polarization datasets hindering learning-based approaches.
Approach
Transfer an RGB-D pre-trained depth completion model to polarization input by inserting a learnable Polarization Prompt Fusion Block (parallel spatial attention-like and channel fovea branches) at every encoder layer, so polarization acts as a prompt that is progressively fused into pre-trained features.
Key results
- On HAMMER (all sensors), the method reaches RMSE 28.63 mm and MAE 14.17 mm, versus 38.88 mm and 19.94 mm for CompletionFormer.
- Replacing polarization with RGB in the same PPFT architecture raises RMSE to 33.06 mm; without PPFT (naive polarization) RMSE is 41.17 mm.
- On shape-from-polarization (SPW), iDisc + PPFT lowers mean angular error from 12.99 to 10.91 degrees.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PPFT with polarization guidance instead of RGB guidance (ablation) | PPFT depth enhancement · HAMMER All sensors | RMSE: 33.06 -> 28.63 | Helped |
| use PPFT with polarization guidance instead of RGB guidance (ablation) | PPFT depth enhancement · HAMMER All sensors | MAE: 16.55 -> 14.17 | Helped |
| add PPFT cross-modal transfer learning (ablation) | polarization-guided depth enhancement · HAMMER All sensors | RMSE: 41.17 -> 28.63 | Helped |
| add PPFT cross-modal transfer learning (ablation) | polarization-guided depth enhancement · HAMMER All sensors | MAE: 22.14 -> 14.17 | Helped |
| replace naive early fusion with PPFB fusion (ablation) | polarization-guided depth enhancement · HAMMER All sensors | RMSE: 30.33 -> 28.63 | Helped |
| replace naive early fusion with PPFB fusion (ablation) | polarization-guided depth enhancement · HAMMER All sensors | MAE: 15.30 -> 14.17 | Helped |
| extend PPFB from the first encoder layer to deep layers (ablation) | PPFB-shallow depth enhancement · HAMMER All sensors | RMSE: 29.77 -> 28.63 | Helped |
| extend PPFB from the first encoder layer to deep layers (ablation) | PPFB-shallow depth enhancement · HAMMER All sensors | MAE: 14.79 -> 14.17 | Helped |
| add polarization guidance rather than use no polarization (ablation) | depth enhancement baseline [65] · HAMMER All sensors | RMSE: 43.64 -> 28.63 | Helped |
| add polarization guidance rather than use no polarization (ablation) | depth enhancement baseline [65] · HAMMER All sensors | MAE: 22.84 -> 14.17 | Helped |
| apply PPFT to iDisc for shape-from-polarization (ablation) | iDisc · SPW | Mean angular error: 12.99 -> 10.91 | Helped |
| apply PPFT to iDisc for shape-from-polarization (ablation) | iDisc · SPW | Med angular error: 10.11 -> 8.92 | Helped |
Surprising findings
- Naively passing polarization into RGB-guided depth completion models yields little gain (RMSE 43.64 to 41.17 mm on HAMMER).
- The Polarized 3D physics-based baseline fails in this setting when run with author-provided code (RMSE above 700 mm).
Limitations
- Evaluated on a single public dataset (HAMMER) with limited scenes.
- Relies on pre-trained RGB-D depth completion weights; benefit depends on that transfer.
- Physics-based Polarized 3D baseline could not be run successfully with released code.
Experimental setup
Datasets: HAMMER NYU-Depth V2 SPW
Benchmarks: HAMMER SPW
Metrics: RMSE MAE delta1 delta2 delta3 mean angular error median angular error percentage within 11.5/22.5/30 degrees
Baselines: Polarized 3D SemAttNet DySPN CompletionFormer Early Fusion iDisc SfP-Wild
Builds on: CompletionFormer Spatial fovea Channel fovea (Li et al.) Polarized 3D
Closest papers by meaning
- Depth Prompting for Sensor-Agnostic Depth Estimation (CVPR 2024 · similarity 0.89)
- PolarDepth: Monocular Transparent Object Depth from Polar-Physics Priors (ICML 2026 · similarity 0.88)
- RGB-to-Polarization Estimation: A New Task and Benchmark Study (NeurIPS 2025 · similarity 0.85)
- PolarFree: Polarization-based Reflection-Free Imaging (CVPR 2025 · similarity 0.84)
- Glossy Object Reconstruction with Cost-effective Polarized Acquisition (CVPR 2025 · similarity 0.83)
- SVDC: Consistent Direct Time-of-Flight Video Depth Completion with Frequency Selective Fusion (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.