Research map / Image generation / Image restoration
Intra and Inter Parser-Prompted Transformers for Effective Image Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PPTformer, a Transformer for image restoration that injects SAM-generated parser features via intra/inter parser-prompted attention and a parser-prompted feed-forward network, evaluated on deraining, defocus deblurring, desnowing, and low-light enhancement.
Problem
Image restoration relies only on degraded inputs, which often contain unreliable pixels; the paper asks whether structural parses from a visual foundation model (SAM) can help restoration and how to integrate them.
Approach
Uses SAM-generated hierarchical parses, encoded by an autoencoder-like PPFGNet, to prompt a Transformer restoration network through intra-layer cross-attention (implicit), fused inter-attention (explicit), a parser-gated feed-forward network, and a bidirectional parser-prompted fusion (BiPPF) module that replaces SFT.
Key results
- On deraining, averaged over Test100, Rain100H, Test2800, PPTformer reaches 32.42 dB PSNR, about 0.07 dB above SFNet.
- On LOL and LOL-v2 low-light enhancement, PPTformer reports 25.48 dB and 23.52 dB PSNR, 2.39 dB and 2.74 dB above UHDFour on LOL and LOL-v2 respectively.
- On defocus deblurring (DPDD combined), PPTformer gets 26.13 dB PSNR, with Restormer roughly comparable (25.98 dB) and a 0.15 dB gain over Restormer on combined scenes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PPTformer instead of SFNet | Image deraining · Image deraining Average | PSNR: 32.35 -> 32.42 | Helped |
| use PPTformer instead of Restormer | Single-image defocus deblurring · DPDD test, Combined | PSNR: 25.98 -> 26.13 | Helped |
| use PPTformer instead of Restormer | Image desnowing · CSD (2000) | PSNR: +0.84 dB | Helped |
| use PPTformer instead of Restormer | Image desnowing · SRRS (2000) | PSNR: +1.25 dB | Helped |
| use PPTformer instead of Restormer | Image desnowing · Snow100K (2000) | PSNR: +1.47 dB | Helped |
| use PPTformer instead of UHDFour | Low-light enhancement · LOL | PSNR: +2.39 dB | Helped |
| use PPTformer instead of UHDFour | Low-light enhancement · LOL-v2 | PSNR: +2.74 dB | Helped |
| add SAM parser guidance (ablation) | PPTformer · LOL, 100,000 iterations | SSIM: 0.9153 -> 0.9248 | Helped |
| add SAM parser guidance (ablation) | PPTformer · LOL, 100,000 iterations | LPIPS: 0.1211 -> 0.1138 | Helped |
| use SAM parser instead of degraded-image guidance (ablation) | PPTformer · LOL, 100,000 iterations | SSIM: 0.9207 -> 0.9248 | Helped |
| add IntraPPA and InterPPA (ablation) | PPTformer · LOL, 100,000 iterations | SSIM: 0.9110 -> 0.9248 | Helped |
| use the IntraPPA-InterPPA structure instead of two IntraPPA modules (ablation) | PPTformer · LOL, 100,000 iterations | SSIM: 0.9176 -> 0.9248 | Helped |
Surprising findings
- Parser-prompting with degraded images as conditions gave lower SSIM/LPIPS than with SAM parses, but the degraded-image variant used more FLOPs and parameters, which the paper notes.
- SAM is run offline and its parses are fed as images, so SAM intermediate features are not used during training.
Limitations
- Uses offline SAM parses rather than end-to-end SAM features, which the authors say may leave performance on the table.
- Effectiveness shown only on low-light enhancement in ablations; untested on dynamic deblurring and dehazing.
- Gains over strong baselines are small on some tasks (e.g., defocus deblurring comparable to Restormer, lower SSIM/LPIPS than Restormer on some metrics).
Experimental setup
Datasets: LOL LOL-v2 DPDD CSD SRRS Snow100K Test100 Rain100H Test2800
Benchmarks: Test100 Rain100H Test2800 DPDD CSD SRRS Snow100K LOL LOL-v2
Metrics: PSNR SSIM MAE LPIPS FLOPs Params
Baselines: Restormer Uformer MAXIM-2S SFNet MPRNet UHDFour URetinex-Net MSP-Former TransWeather IFAN
Builds on: Restormer GDFN SFT SAM
Closest papers by meaning
- Boosting Image Restoration via Priors from Pre-trained Models (CVPR 2024 · similarity 0.88)
- Sharing Key Semantics in Transformer Makes Efficient Image Restoration (NeurIPS 2024 · similarity 0.87)
- Controlling Vision-Language Models for Multi-Task Image Restoration (ICLR 2024 · similarity 0.87)
- InstructRestore: Region-Customized Image Restoration with Human Instructions (NeurIPS 2025 · similarity 0.86)
- Dual Prompting Image Restoration with Diffusion Transformers (CVPR 2025 · similarity 0.86)
- Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.