Research map / Image generation / Image restoration
Distilling Semantic Priors from SAM to Efficient Image Restoration Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills SAM-derived semantic priors into existing image restoration models via a cascaded self-distillation framework (SPF fusion, SPD distillation with a semantic-guided relation module), so SAM is used only at training time.
Problem
Using SAM semantic priors for image restoration improves quality but adds substantial SAM inference cost at test time, hurting efficiency of IR models.
Approach
A second IR network fuses the first network's restored image with SAM masks via SPF units, and the first network is trained to match the refined output through smooth-L1 distillation plus a VGG-based semantic relation loss over mask-guided features; only the first network is used at inference.
Key results
- Framework improves RCDNet by 0.91 dB PSNR on average and Efficientderain by 1.02 dB PSNR across deraining datasets.
- On GoPro deblurring Uformer improves from 32.10 to 32.21 PSNR; on SenseNoise denoising from 35.14 to 35.25 PSNR.
- Ablation on Rain200L shows SGR with SAM reaches 35.29 PSNR for f_IR1 versus 34.09 for the baseline IR model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply the proposed framework to RCDNet | RCDNet · Rain200L, Rain200H, DID, and DDN | PSNR: +1.22; +0.58; +0.55; +0.77 | Helped |
| Apply the proposed framework to Efficientderain | Efficientderain · Rain200L, Rain200H, DID, and DDN | PSNR: +1.28; +1.11; +0.73; +0.73 | Helped |
| Apply the proposed framework to RCDNet | RCDNet · Cityscape-syn 100 mm and Cityscape-syn 200 mm | PSNR: +1.16; +1.19 | Helped |
| Apply the proposed framework to Efficientderain | Efficientderain · Cityscape-syn 100 mm and Cityscape-syn 200 mm | PSNR: +1.23; +1.25 | Helped |
| Apply the proposed framework to Uformer | Uformer · GoPro | PSNR: +0.11 | Helped |
| Apply the proposed framework to Uformer | Uformer · SenseNoise | PSNR: +0.11 | Helped |
| Add cascaded fIR2 (ablation) | fIR1 · Deraining ablation setting | PSNR | Hurt |
| Add SPF with SAM priors (ablation) | fIR1 + fIR2 · Deraining ablation setting | PSNR | Helped |
| Add SPD with SAM priors (ablation) | fIR1 + fIR2 + SPF · Deraining ablation setting | PSNR | Helped |
| Add the SGR module with SAM priors (ablation) | fIR1 + fIR2 + SPF + SPD · Deraining ablation setting | PSNR | Helped |
| Use SAM rather than PSPNet semantic priors with SGR (ablation) | fIR1 + fIR2 + SPF + SPD + SGR · Deraining ablation setting | PSNR | Helped |
| Set λ1 to 0.0005 instead of 0.005 (ablation) | Proposed framework · Deraining ablation setting | PSNR | Hurt |
Surprising findings
- SAM-derived priors gave larger gains than PSPNet priors in ablation, yet the PSPNet variant still improved over baseline.
Limitations
- Gains are modest on deblurring and denoising (about 0.1 dB).
- Training requires running SAM and a cascade of two IR networks, increasing training cost.
Experimental setup
Datasets: Rain200L Rain200H DID DDN Cityscape-syn100 Cityscape-syn200 GoPro SenseNoise
Benchmarks: Rain200L Rain200H DID DDN Cityscape-syn100 Cityscape-syn200 GoPro SenseNoise
Metrics: PSNR SSIM FID IoU PA DICE
Baselines: RCDNet Efficientderain Uformer RESCAN PReNet MSPFN MPRNet DeblurGAN DMPHN CBDNet UNet
Builds on: Segment Anything Model RCDNet Efficientderain Uformer
Closest papers by meaning
- UniRestore: Unified Perceptual and Task-Oriented Image Restoration Model Using Diffusion Prior (CVPR 2025 · similarity 0.89)
- Navigating Image Restoration with VAR's Distribution Alignment Prior (CVPR 2025 · similarity 0.86)
- URWKV: Unified RWKV Model with Multi-state Perspective for Low-light Image Restoration (CVPR 2025 · similarity 0.86)
- ReFIR: Grounding Large Restoration Models with Retrieval Augmentation (NeurIPS 2024 · similarity 0.86)
- Boosting Image Restoration via Priors from Pre-trained Models (CVPR 2024 · similarity 0.85)
- Improving Image Restoration through Removing Degradations in Textual Representations (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.