Research map / Image generation / Image restoration
RGB-to-Polarization Estimation: A New Task and Benchmark Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RGB-to-polarization (Stokes) estimation as a new task and benchmarks restoration-based (Restormer, Uformer, MAE) and generative (WDiff, DiT, RealFill, Img2ImgTurbo) models on the Jeon et al. RGB-polarization dataset.
Problem
Polarization images carry physical cues absent from RGB but require specialized, costly hardware; the paper asks whether polarization (Stokes S1-S3) can be estimated directly from a single RGB image.
Approach
Frames sensor-free polarization imaging as a pixel-wise RGB-to-Stokes regression/generation task and establishes the first benchmark comparing restoration backbones, task-specific diffusion models, and pre-trained (MAE, Stable Diffusion) adaptations under a standardized protocol.
Key results
- MAE achieves the best average PSNR 24.74 and SSIM 0.8876 on the Jeon et al. test split, while Uformer has the lowest average LPIPS 0.2267.
- Restoration models (Restormer, Uformer, MAE) generally outperform diffusion models; WDiff is weakest with average PSNR 13.11.
- Replacing all real polarization channels with estimated ones in ShareCMP segmentation drops mIoU from 92.45 to 45.56.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MAE pretrained weights and fine-tune end-to-end instead of training Uformer from scratch | RGB-to-polarization estimation · Jeon test set | PSNR: 24.34 -> 24.74 | Helped |
| Use MAE pretrained weights and fine-tune end-to-end instead of training Uformer from scratch | RGB-to-polarization estimation · Jeon test set | SSIM: 0.8722 -> 0.8876 | Helped |
| Use Uformer rather than MAE | RGB-to-polarization estimation · Jeon test set | LPIPS: 0.2684 -> 0.2267 | Helped |
| Replace WDiff with DiT | Task-specific diffusion RGB-to-polarization estimation · Jeon test set | PSNR: 13.11 -> 23.33 | Helped |
| Replace RealFill with Img2ImgTurbo | Pretrained diffusion RGB-to-polarization estimation · Jeon test set | PSNR: 21.81 -> 23.33 | Helped |
| Replace RealFill with Img2ImgTurbo | Pretrained diffusion RGB-to-polarization estimation · Jeon test set | SSIM: 0.8025 -> 0.8725 | Helped |
| Use restoration-based models rather than diffusion models | RGB-to-polarization estimation · Jeon test set | PSNR | Mixed |
| Adapt Depth Anything v2 pretrained weights to predict polarization | RGB-to-polarization estimation · Jeon test set | PSNR: 24.74 -> 25.75 | Helped |
| Adapt Depth Anything v2 pretrained weights to predict polarization | RGB-to-polarization estimation · Jeon test set | LPIPS: 0.2684 -> 0.3765 | Hurt |
| Add a DoP range penalty enforcing DoP within [0,1] (ablation) | RGB-to-polarization estimation models · Polarization estimation | perceptual and polarization-related metrics | Helped |
| Replace real polarization inputs with estimated polarization inputs | ShareCMP segmentation on UPLight · UPLight | mIoU: 92.45 -> 45.56 | Hurt |
| Estimate only I45 while keeping the other polarization inputs real | ShareCMP segmentation · UPLight | mIoU: 92.45 -> 92.40 | No clear effect |
Surprising findings
- Img2ImgTurbo has higher PSNR than DiT yet produces visually less faithful structure, indicating a trade-off between pixel metrics and perceptual quality.
- Depth Anything v2 adapted to output Stokes achieves higher PSNR than MAE but worse perceptual quality (over-smoothing).
Limitations
- Evaluation limited to one main dataset with restricted material diversity (metals, birefringent materials, tissues underexplored).
- No adverse conditions (weather, extreme noise) evaluated.
- Physical constraints (DoP range, Stokes consistency) are only briefly explored and not part of main benchmark.
Experimental setup
Datasets: Jeon et al. RGB-polarization dataset Qiu et al. Kurita et al.
Benchmarks: Jeon et al. RGB-polarization dataset Qiu et al. Kurita et al. UPLight (downstream segmentation)
Metrics: PSNR SSIM LPIPS mIoU
Baselines: Restormer Uformer WDiff DiT RealFill Img2ImgTurbo Depth Anything v2
Builds on: Restormer Uformer MAE Stable Diffusion RealFill
Closest papers by meaning
- PolarScale: A Physics-Grounded Benchmark for Radiometrically Consistent RGB-to-Stokes Estimation (NeurIPS 2026 · similarity 0.90)
- PolarFree: Polarization-based Reflection-Free Imaging (CVPR 2025 · similarity 0.89)
- Glossy Object Reconstruction with Cost-effective Polarized Acquisition (CVPR 2025 · similarity 0.88)
- Spectral and Polarization Vision: Spectro-polarimetric Real-world Dataset (CVPR 2024 · similarity 0.87)
- Exploiting Polarized Material Cues for Robust Car Detection (AAAI 2024 · similarity 0.85)
- DRIP: Unleashing Diffusion Priors for Joint Foreground and Alpha Prediction in Image Matting (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.