Research map / Label-efficient and robust learning / Self-supervised and contrastive
Transfer CLIP for Generalizable Image Denoising
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Transfers the frozen ResNet-50 CLIP image encoder into an asymmetrical encoder-decoder denoiser, trained on one Gaussian noise level, to generalize to out-of-distribution noise, with progressive feature augmentation.
Problem
Image denoising models trained on one noise type/level generalize poorly to out-of-distribution (OOD) noise, including synthetic, real-world sRGB, and low-dose CT noise.
Approach
Dense multi-scale features from the frozen ResNet-50 CLIP image encoder are distortion-invariant and content-related, so a learnable convolutional decoder fed with these features and the noisy image can denoise robustly; progressive feature augmentation (multiplicative Gaussian noise increasing with scale) reduces feature overfitting.
Key results
- On Gaussian sigma=50 (OOD) on CBSD68 the method reaches 26.69 dB vs 26.43 for DIL and 20.68 for MaskDenoising, while being trained only at sigma=15.
- On real-world sRGB, it reaches 37.54/0.960 on PolyU and 36.30/0.941 on CC, beating CVF-SID and LUD-VAE on those datasets.
- On low-dose CT, it reaches 44.79/0.970 vs 44.71/0.971 for Noise2Sim and 43.55/0.960 for ScoreSDE, without needing adjacent LDCT images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CLIPDenoising rather than the strongest listed baseline | denoising on Kodak24 · Gaussian σ = 50 | PSNR: 27.46 -> 27.39 | Hurt |
| use CLIPDenoising rather than the strongest listed baseline | denoising on Kodak24 · spatial Gaussian σ = 55 | PSNR: 27.28 -> 28.29 | Helped |
| use CLIPDenoising rather than the strongest listed baseline | denoising on Kodak24 · Poisson α = 3.5 | PSNR: 28.60 -> 28.66 | Helped |
| use CLIPDenoising rather than the strongest listed baseline | denoising on Kodak24 · Salt&Pepper d = 0.02 | PSNR: 30.56 -> 30.61 | Helped |
| use CLIPDenoising rather than the strongest listed baseline | real-world sRGB denoising · SIDD Val | PSNR: 34.91 -> 34.79 | Hurt |
| use CLIPDenoising rather than the strongest listed baseline | real-world sRGB denoising · PolyU | PSNR: 37.65 -> 37.54 | Hurt |
| use CLIPDenoising rather than the strongest listed baseline | real-world sRGB denoising · CC | PSNR: 36.10 -> 36.30 | Helped |
| use CLIPDenoising rather than Noise2Sim | low-dose CT denoising · Low-dose CT abdomen slices | PSNR: 44.71 -> 44.79 | Helped |
| add the noisy input to the decoder (ablation) | CLIPDenoising · Kodak24, Gaussian σ = 50 | PSNR: 21.76 -> 26.87 | Helped |
| remove feature F5 (ablation) | CLIPDenoising · Kodak24, Gaussian σ = 50 | PSNR: 25.61 -> 26.87 | Helped |
| keep the CLIP RN50 encoder frozen instead of random initialization (ablation) | CLIPDenoising · Kodak24, Gaussian σ = 50 | PSNR: 20.87 -> 26.87 | Helped |
| keep the CLIP RN50 encoder frozen instead of fine-tuning it (ablation) | CLIPDenoising · Kodak24, Gaussian σ = 50 | PSNR: 19.03 -> 26.87 | Helped |
Surprising findings
- Fine-tuning the CLIP encoder or training it from scratch improves in-distribution PSNR but destroys OOD robustness.
- MoCo-v3 ResNet-50 features also show some OOD robustness, suggesting the property is not unique to CLIP.
- Larger CLIP ResNets (RN50x4, RN50x16) lose the distortion-invariant shallow features and generalize worse.
Limitations
- Slightly underperforms HAT and DIL on speckle noise and does not consistently beat them on every OOD type.
- Transformer-based decoder overfits training noise and loses generalization; ViT CLIP encoder unsuitable due to downsampling.
- Distortion-invariant property is specific to shallow RN50 features and does not hold for larger ResNets.
Experimental setup
Datasets: CBSD432 DIV2K AAPM-Mayo Low Dose CT Grand Challenge Kodak24 McMaster CBSD68 Urban100 SIDD PolyU CC
Benchmarks: CBSD68 McMaster Kodak24 Urban100 SIDD validation PolyU CC AAPM-Mayo LDCT (patient L506)
Metrics: PSNR SSIM
Baselines: DnCNN Restormer MaskDenoising HAT DIL CVF-SID LUD-VAE Noise2Sim ScoreSDE-CT TV
Builds on: CLIP MaskDenoising DIL HAT ScoreSDE
Closest papers by meaning
- Robust And Interpretable Blind Image Denoising Via Bias-Free Convolutional Neural Networks (ICLR 2020 · similarity 0.87)
- Random Is All You Need: Random Noise Injection on Feature Statistics for Generalizable Deep Image Denoising (ICLR 2025 · similarity 0.87)
- Fully Unsupervised Diversity Denoising with Convolutional Variational Autoencoders (ICLR 2021 · similarity 0.86)
- Toward Convolutional Blind Denoising of Real Photographs (CVPR 2019 · similarity 0.86)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.85)
- Zero-Shot Noise2Mean: Gap Minimization for Efficient Denoising from a Single Noisy Image (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.