Research map / Image generation / Image restoration
Optimal Eye Surgeon: Finding image priors through sparse generators at initialization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Optimal Eye Surgeon (OES), which learns a Bernoulli-relaxed (Gumbel-softmax) mask over an untrained U-Net at random initialization so the resulting sparse subnetwork ("Sparse-DIP") acts as an image prior that resists overfitting to noise in denoising.
Problem
Deep image prior (DIP) overfits to noise in image restoration because it is heavily overparameterized, while underparameterized deep decoders oversmooth fine detail; the question is whether a sparse subnetwork can combine good priors with resistance to overfitting, and how to find it without the cle
Approach
Learn a sparsity mask at random initialization by minimizing the expected fit loss to the single corrupted image under Bernoulli masks, with a KL penalty toward a sparsity-matched prior, optimized via Gumbel-softmax (Concrete) relaxation; then train only the masked subnetwork. The mask needs no clean image, unlike LTH-style pruning.
Key results
- Sparse-DIP (3% sparse, ~90k params) outperforms Dense DIP, GP-DIP and Deep Decoder on most Set-14 images at sigma=25 dB without early stopping (e.g., Pepper 27.45 vs 27.01 for Deep Decoder).
- At 5% sparsity, OES outperforms GraSP, SynFlow, magnitude, random pruning and IMP on most images; e.g., Face-2 29.58 vs 26.19 for IMP-(0.2)^3.
- Masks learned from one image transfer to other images, including across datasets, with OES masks outperforming IMP masks in transfer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prune the DIP at initialization with OES and train the resulting 3% sparse network | DIP · Pepper, σ=25dB | PSNR: 21.21 -> 27.45 | Helped |
| Use OES pruning rather than the strongest listed pruning baseline | DIP · Pepper, σ=25dB, 5% sparsity | PSNR: 25.66 -> 27.45 | Helped |
| Use OES-pruned Sparse-DIP rather than Deep Decoder | image denoising · Pepper, σ=25dB | PSNR: 27.01 -> 27.45 | Helped |
| Use OES-pruned Sparse-DIP rather than Deep Decoder | image denoising · Face-3, σ=25dB | PSNR: 28.27 -> 27.91 | Hurt |
| Use OES mask learned from the target image rather than a transferred mask | OES Sparse-DIP · cross-image denoising | PSNR | Helped |
| Use a transferred OES mask rather than an IMP mask | image denoising · inter-dataset and cross-dataset transfer | PSNR | Helped |
| Increase sparsity from 3% to 80% (ablation) | OES Sparse-DIP · image denoising | overfitting | Hurt |
| Replace OES mask learning with SynFlow pruning at initialization | DIP · image denoising | PSNR: 22.07 -> 27.45 | Hurt |
| Replace OES mask learning with GraSP pruning at initialization | DIP · image denoising | PSNR: 22.22 -> 27.45 | Hurt |
| Use magnitude pruning at initialization instead of OES | DIP · Pepper, σ=25dB, 5% sparsity | PSNR: 12.42 -> 27.45 | Hurt |
| Use random pruning at initialization instead of OES | DIP · Pepper, σ=25dB, 5% sparsity | PSNR: 10.80 -> 27.45 | Hurt |
| Use gradual IMP pruning rather than aggressive IMP pruning (ablation) | DIP · image denoising | PSNR: 25.66 vs 25.52 | Helped |
Surprising findings
- Masking a random-initialized network with OES masks alone (no training) already yields a low-frequency image approximation, i.e., the strong lottery ticket effect holds for image reconstruction.
- LTH masks derived from converged DIP magnitudes are detrimental for denoising, producing overfitted subnetworks.
Limitations
- Sparse networks overfit slightly when transferred across different domains.
- Mask learning adds computational overhead due to Gumbel-softmax reparameterization.
- Masks are architecture-specific, so transfer across tasks with different architectures (e.g., two-channel U-Net for MRI) is limited.
Experimental setup
Datasets: Set-14 standard image dataset Face dataset
Benchmarks: Set-14 standard image dataset Face dataset
Metrics: PSNR
Baselines: Dense DIP GP-DIP Deep Decoder GraSP SynFlow Magnitude pruning Random pruning IMP
Builds on: Deep Image Prior Deep Decoder Lottery Ticket Hypothesis Gumbel-softmax GraSP SynFlow
Closest papers by meaning
- Learnable Sparsity for Vision Generative Models (ICLR 2026 · similarity 0.87)
- The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information (NeurIPS 2024 · similarity 0.86)
- Preserving Deep Representations in One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework (ICLR 2025 · similarity 0.86)
- Adaptive Sharpness-Aware Pruning for Robust Sparse Networks (ICLR 2024 · similarity 0.86)
- OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization (ICML 2024 · similarity 0.85)
- SequentialAttention++ for Block Sparsification: Differentiable Pruning Meets Combinatorial Optimization (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.